From esteban@ccpgames.com Tue Oct 1 05:54:46 2002 From: esteban@ccpgames.com (Esteban U.C.. Castro) Date: Tue, 1 Oct 2002 04:54:46 -0000 Subject: [Python-Dev] proposal for interfaces Message-ID: <25A39AFEB31B06408C675CEF28199E5B073CAA@postur.ccp.cc> In case you have missed the link (I did, at first); this is Guido's opinion on the interfaces PEP as of March 7, 2002. http://www.python.org/doc/essays/pepparade.html Jim Fulton has said that this PEP was premature. I agree. It introduces a new keyword, 'interface', and I'm not yet convinced that that is needed. On the other hand, the way this is currently done in Zope also looks butt-ugly, so something may indeed be needed. I think that at some point in the future when we have more experience with using interfaces (especially in Zope 3) we'll go back to this PEP and see how much of it we can use. Maybe there should be a special status "frozen" meaning not rejected but also not under consideration for the near future? But with a different meaning than Py-in-the-sky -- this PEP at least has lots of concrete proposals and studies the consequences.=20 From martin@v.loewis.de Tue Oct 1 07:07:22 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: Tue, 1 Oct 2002 08:07:22 +0200 Subject: [Python-Dev] Dropping support for Tcl 8.0 and 8.1 Message-ID: <200210010607.g9167Mns001471@mira.informatik.hu-berlin.de> In _tkinter.c, there is currently special code for Tcl 8.0 (no Tcl objects) and Tcl 8.1 (no Unicode type). Tcl 8.2, 8.3, and 8.4 are supported uniformly. Can we drop support for those two releases, in Python 2.3? Regards, Martin From drifty@bigfoot.com Tue Oct 1 08:44:09 2002 From: drifty@bigfoot.com (Brett Cannon) Date: Tue, 1 Oct 2002 00:44:09 -0700 (PDT) Subject: [Python-Dev] Python-dev Summary for 2002-09-16 to 2002-09-30 Message-ID: Once again, here is the rough draft for the next Summary. Usual review deal (24 hours, blah blah). Have fun. ======================================== `flextype.c -- extended type system`__ ======================================== __ http://mail.python.org/pipermail/python-dev/2002-September/028736.html (continuation from `last summary`_) To clarify what Christian Tismer's metatype patch basically implements, he said that his "additions support a subset of C++ virtual methods". And in response to a question from me about how this might affect magic methods, he said he "*could* support any magic slot and put it into the extended type object with a Python name. And even better, this version could have full type checking, as my other methods have as well!" Martin v. Loewis doesn't see the use of this because extensions in C are usually just wrappers and C++ extensions already have the functionality. Thomas Heller, though, likes it because he wants a way "to add additional C accessible structure fields to types. Currently [he is] attaching them to the the type's dict, as I reported before" by David Abrahams. As an aside, a suggestion to David about having to put instance-specific info in __dict__, Thomas said "You can (but you probably know this already) replace the type's tp_dict by a custom subclass of PyDict_Object, which adds additional fields." .. _last summary: http://www.ocf.berkeley.edu/python-dev/summaries/2002-09-01--2002-09-15.html ================================== `Moratorium on changes to IDLE`__ ================================== __ http://mail.python.org/pipermail/python-dev/2002-September/028776.html Guido said that he would "like to put a stop to all changes to the version of IDLE in the Python source tree (`Tools/idle/*`_ -- let's call it Python-idle)". All changes should go through IDLEfork_ . It will then get merged into CVS before 2.3 is released. Currently Idlefork is working with 2.2 and Guido would like to keep it that way so that more people use IDLE. .. _Tools/idle/*: .. _IDLEfork: =========================================== `Weeding out obsolete modules and Demos`__ =========================================== __ http://mail.python.org/pipermail/python-dev/2002-September/028718.html (continuation from `last summary`_) Guido removed the obsolete SGI demos from Demo_ as mentioned from the last summary. If anyone else out there has any demos that they think are no longer needed, submit a `SF patch`_ mentioning what you think should be removed and why. .. _Demo: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Demo/ .. _SF patch: http://sourceforge.net/tracker/group_id=5470&atid=305470 ======================= `ReST-ing Misc/NEWS`__ ======================= __ http://mail.python.org/pipermail/python-dev/2002-September/028799.html Skip changed `Misc/NEWS`_ into a reST document. Hopefully this will end up somewhere on http://www.python.org so people can be kept very up-to-date on what changes have been made in CVS_ . .. _Misc/NEWS: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Misc/NEWS .. _CVS: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/ ==================================== `mysterious hangs in socket code`__ ==================================== __ http://mail.python.org/pipermail/python-dev/2002-September/028555.html (continued from `last summary`_) Still trying to weed out issues with threaded code that seems to act single-threaded. The suspect is still glibc's getaddrinfo() implementation; POSIX says it is supposed to be thread-safe but not everyone is believing that. ===================================================== `ATTENTION! Releasing Python 2.2.2 in a few weeks`__ ===================================================== __ http://mail.python.org/pipermail/python-dev/2002-September/028806.html Guido announced that he would like to get Python 2.2.2 out the door by October 8. PythonLabs is busy as it is and thus do not have time to thoroughly make sure everything gets done. What can you do to help? Glad you asked: "the best thing you can do is take your favorite set of modules or core files and systematically backport anything that's clearly a bugfix and backports easily. Or you could simply make sure that your favorite bugfix is backported". If you don't have checkin, either team up with someone who does or request checkin rights (decision will be based on reputation). SF should not be used for saying what should and shouldn't be backported. This should all be done through Python-dev. Michael McLay suggest a wiki to handle all of this (thank goodness; the amount of traffic this would have generated would have killed me). Skip said he could set one up rather quickly and he did at http://manatee.mojam.com/py222wiki/ . Laura Creighton mentioned what she planned to do for Python-in-a-Tie (Py-tie). Looks like the current plan is to make Python 2.2.2 the initial release and add in things from there based on developer feedback instead of taking future releases and removing what they don't need. Perhaps Py-tie will end up taking the place of the elusive `Sumo distro`_? Ralf W. Grosse-Kunstleve brought up how a bug report about C++ compatibility had not been handled yet. Both Tim and Guido said that if he wanted a solution put in they would need a patch; neither of them are C++ users. C++ uses should keep that in mind when trying to get C++ stuff into the core. Gustavo Niemeyer said that he has a script for adding moin pages using your favorite editor. He also has a syntax highlight file for Vim. All of this can be found at http://moin.conectiva.com.br/EditMoin . .. _Sumo distro: http://www.python.org/peps/pep-0206.html ================================ `Built-in functions, kw args`__ ================================ __ http://mail.python.org/pipermail/python-dev/2002-September/028811.html Jp Calderone brought up the question of whether ParseTupleAndKeywords should be used more often then ParseTuple in the core. Guido pointed out that ParseTupleAndKeywords did not exist since the beginning of time, it's slower, and much harder to use. Guido basically ended up saying use it if you want, it isn't required. ======================= `Python for OpenVMS`__ ======================= __ http://mail.python.org/pipermail/python-dev/2002-September/028814.html Skip Montanaro announced that Jean-Fran?ois Pi?ronne had ported Python 2.1.3 over to OpenVMS_ and had begun work on porting the 2.3 tree. .. _OpenVMS: http://www.openvms.org/ ================ `-zcombreloc`__ ================ __ http://mail.python.org/pipermail/python-dev/2002-September/028822.html In Andrew Koenig's continual quest to get Python to build with Binutils 2.13 under Solaris, Andrew discovered through discussions with the Binutils developers "that -zcombreloc just plain doesn't work in any release of binutils". Andrew thinks this will get fixed in Binutils 2.13.1 . But what about until then? Well, you are out of luck on Solaris with Binutils 2.13 . There is a test you can run to see if you will be affected by this bug in this email (link to email). ======================================= `binutils/solaris -- one more thing`__ ======================================= __ http://mail.python.org/pipermail/python-dev/2002-September/028826.html Andrew Koenig discovered another issue with Solaris/Binutils. In Binutils 2.12, the output from ``ld -V`` which is used to find out if dynamic linking is available changed and broke the test. It turns out Martin v. Loewis already committed a patch to Python 2.2.2 and 2.3 that deals with this issue. =================================================== `Need advice: cloning python cvs for CE project`__ =================================================== __ http://mail.python.org/pipermail/python-dev/2002-September/028842.html Brad Clements of the "python ce" project asked for advice on how to "maintain a single working directory that can be checked into two different CVS systems". Guido said they could have a branch in the CVS tree if they were all Python developers "or can be sworn in easily". The other option Guido presented was for them to start their own SF project and do a vendor branch checking; this is how IDLE-fork does it. Martin gave some instructions on how to do this in an email which can be reached at XXX . ============================= `Assign to errno allowed?`__ ============================= __ http://mail.python.org/pipermail/python-dev/2002-September/028868.html Thomas Heller was patching selectmodule.c_ for Windows to deal with the fact that it does not set errno. Thomas wondered if it would be okay to set errno himself. Everyone in the end said it was fine (even the man page for errno). Through the discussion over whether not setting errno was proper, it was pointed out by Thomas that the entire reason he was fixing this was for asyncore_ to work; apparently it has always been broken on Windows. Guido suggested a simple fix to the module to make it work. The other point that came up that Windows CE and NetWare don't allow for direct signing to errno; ``errno = 0`` is an error on those platforms. Brad Clements submitted a patch to add a macro to Python for setting errno. Well, Tim pointed out that it is an ISO C requirement that errno be assignable. This led Guido to "have a very strong urge to tell [Brad's patch] to go away". Let me say that from my personal experience on Python-dev I have noticed that Python-dev does not like work-arounds for things that don't work because something does not follow a standard; this is especially true for anything dealing with ISO C. So if you have a problem and it is because your platform is not standards-compliant, I would not expect to get much sympathy from Python-dev (unless it is a Windows desktop OS =). In the end it was suggested to the CE guys to just keep a branch of the CVS where they fixed the whole errno issue. That way they can just sync with the core and not have to go through and change all of the errno lines every time they sync. This thread even split into another one entitled `Reserved keywords in source`_ in which Brad Clements pointed out that ``finally`` is a reserved word on Windows CE. This, once again, flies in the face of ISO C. .. _selectmodule.c: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Modules/selectmodule.c .. _asyncore: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Lib/asyncore.py .. _Reserved keywords in source: http://mail.python.org/pipermail/python-dev/2002-September/028912.html ============= `sorted()`__ ============= __ http://mail.python.org/pipermail/python-dev/2002-September/028925.html Francois Pinard brought up the point that most beginners don't realize that ``.sort()`` does not return anything; the sorting is done in-place. He thought that adding a ``.sorted()`` method to lists that would work just like ``.sort()`` but also return the list would be nice. Would not only possibly simplify some code by allowing a sort method to be called in the header of a ``for`` loop, but possibly prevent newbie errors as previously mentioned. Greg Ewing suggested a ``sorted()`` function instead that would basically do what Francois wanted. This causeed Timothy Delaney to suggest possible magic methods ``__sort__`` and ``__isort__``. Guido responded saying that "this is a problem that should not be addressed by changes to the language, builtins or library". So there you go, the BDFL has spoken. ============================== `Lib/plat-xxxx directories`__ ============================== __ http://mail.python.org/pipermail/python-dev/2002-September/028933.html In case people are interested, the Lib/plat-xxx directories are "for platform-specific modules". Most of them, apparently, "are collections of system constants generated by Tools/scripts/h2py.py" . .. _Tools/scripts/h2py.py: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Tools/scripts/h2py.py ================================== `How to add an encoding alias?`__ ================================== __ http://mail.python.org/pipermail/python-dev/2002-September/028936.html For the spambayes_ project, Guido asked how to add an encoding to ``encodings.aliases``. He wondered if there was an official API or if:: import encodings.aliases encodings.aliases.aliases['ansi-x3-4-1968'] = 'ascii' # This is wrong! Read on for why... was the way to do it. M.A. Lemburg pointed responded that the above-mentioned way is the only way to do it in 2.2 . He also mentioned that hyphens must be changed to underscores for names. MAL suggested that all non-alphanumeric characters be mapped to underscores before referring to the aliases dictionary. Guido voted +1 for the change and +1 to backporting to 2.2.2 . .. _spambayes: http://spambayes.sf.net ======================================== `User extendable literal modifiers?!`__ ======================================== __ http://mail.python.org/pipermail/python-dev/2002-September/028945.html M.A. Lemburg asked what everyone thought of having the ability to add new literal modifiers (e.g. ``L`` in ``123L``) to the interpreter. Gareth McCaughan suggest "uder-definable *prefixes*". His example is of Common Lisp's ability to have you define something as ``#`` and then parse it based on . Alex Martelli liked the idea and suggested that ``@`` be the proposed symbol instead of ``$`` as originally suggested. Alex also said that being able to parse multiple tokens would be important. Jeff Epler pointed out that this could all be done right now with some ugly syntax:: class Dimension: ... class DimensionMaker: def __call__(self, v): return Dimension(v) def __add__(self, v): return Dimension(v) D = DimensionMaker() >>> D+"123" # Can be considered same as defining D"123" Guido said that "Given all the discussion, this will need a PEP". He later pointed out that a possible problem is that if you are not very familiar with a program you might probably won't know what a specific prefix is. With a function the worst would be having to look through a module that was imported with ``from import *`` ("that's one reason to avoid these" types of imports). MAL said that Guido was "probably right" about this possible confusion. =========================================== `Keyword for first argument of methods?`__ =========================================== __ http://mail.python.org/pipermail/python-dev/2002-September/028960.html David Abrahams noticed that if he defined the code:: class X: def foo(self, y): print y that ``X.foo(y=1, self=X())`` fails with a TypeError. Guido said that "You can't pass in self to an unbound method as a keyword argument". Best way to have an unbound method is with __call__ . I think using something like the new staticmethod() or classmethod() functions in Python 2.2 would get the job done. ============================================ `Strange bug only happens with Python 2.2`__ ============================================ __ http://mail.python.org/pipermail/python-dev/2002-September/028966.html Gerhard Haring discovered a funky bug dealing with an error saying that a custom object was a string and lacking a __bases__ attribute. What is pertinent to this thread is not the bug (it got fixed in Python 2.3 and has been backported to 2.2.2), but what caused the bug. In abstract.c there was some code that forgot to call PyErr_Clear() . When your extension code catches an error and you decide to either ignore it or go another direction, you must call this function in order to prevent the exception from propogating to the top when the call is finished and raising the exception in the end. ================================================= `builtins instance have modifiable __class__?`__ ================================================= __ http://mail.python.org/pipermail/python-dev/2002-September/028978.html Samuele Pedroni asked if being able to directly assigning to __class__ was allowed. Guido said "In 2.3, it's disallowed". It is highly discouraged in 2.2.2 but still doable. ========================================================== `Why are useful tools omitted from the Win bin distro?`__ ========================================================== __ http://mail.python.org/pipermail/python-dev/2002-September/028988.html Alex Martelli asked why the Windows distribution lacked some of the Tools subdirectory. Most of the scripts are left out since they are meant for developers. Since if you are using the pre-packaged Windows installer chances are you are not a developer since you would have compiled it. Thus these scripts are left out. Tim Peters (who maintains the WISE installer) said that in order to add a script he would have to add an individual line to the installer for each file; a definite PITA (Pain In The Ass) for Tim. There have been calls (which still stand) to document, fix, and clean out the Tools_ and Demo_ directories. No one has ever really stepped up to take this on. Any help, though, no matter how small, would be appreciated in cleaning these directories up. .. _Tools: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Tools/ ======================================= `Extension module difficulty w/pgen`__ ======================================= __ http://mail.python.org/pipermail/python-dev/2002-September/028976.html Jonathan Riehl was having difficulty getting his extension module that exposes pgen to link properly. He got it to work with a little command-line hack; that does not work for the official distro and thus he was searching for a more proper solution. He discovered that if he put dummy calls into a global function that seemed to allow it to be linked. Jonathan hoped there was a better solution. Guido, unfortunately, destroyed that hope by stating "AFAIK you have to create a dummy reference somewhere". He suggested at the end of `Python/pythonrun.c`_ to Jonathan. .. _Python/pythonrun.c: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Python/pythonrun.c ========================= `New logmerge feature`__ ========================= __ http://mail.python.org/pipermail/python-dev/2002-September/028998.html Guido added a new feature to `Tools/logmerge.py`_ that "restricts the output to a specific branch tag". This is useful if you are helping to sort through all of the commits to Python 2.3 to see what should go into 2.2.2 . .. _Tools/logmerge.py: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Tools/logmerge.py ========================================== `Snapshot win32all builds of interest?`__ ========================================== __ http://mail.python.org/pipermail/python-dev/2002-September/029005.html Mark Hammond was wondering if people would be interested in him "making regular, basically untested win32all builds against the current Python CVS tree". If you would like Mark to do this, let him know. From pedronis@bluewin.ch Tue Oct 1 12:48:44 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 1 Oct 2002 13:48:44 +0200 Subject: [Python-Dev] Python-dev Summary for 2002-09-16 to 2002-09-30 References: Message-ID: <006d01c26940$7f3999e0$6d94fea9@newmexico> > ================================================= > `builtins instance have modifiable __class__?`__ > ================================================= > __ http://mail.python.org/pipermail/python-dev/2002-September/028978.html > > Samuele Pedroni asked if being able to directly assigning to __class__ was > allowed. Guido said "In 2.3, it's disallowed". It is highly discouraged > in 2.2.2 but still doable. > my question and the limitation refer only to builtin types (int,dict ...), for instances of (user-defined) subtypes __class__ will be still modifiable. regards. From skip@pobox.com Tue Oct 1 13:41:29 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 1 Oct 2002 07:41:29 -0500 Subject: [Python-Dev] Python-dev Summary for 2002-09-16 to 2002-09-30 In-Reply-To: References: Message-ID: <15769.38905.853668.772022@12-248-11-90.client.attbi.com> Brett> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D Brett> `Python for OpenVMS`__ Brett> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D Brett> __ http://mail.python.org/pipermail/python-dev/2002-Septembe= r/028814.html Brett> Skip Montanaro announced that Jean-Fran?ois Pi?ronne had por= ted Brett> Python 2.1.3 over to OpenVMS_ and had begun work on porting = the Brett> 2.3 tree. Brett> .. _OpenVMS: http://www.openvms.org/ Looks like your text editor or mailer has trouble with accented charact= ers. It's "Jean-Fran=E7ois Pi=E9ronne". ;-) Also, Jean-Fran=E7ois recently s= ubmitted some patches for 2.3 against the code in CVS. Skip From neal@metaslash.com Tue Oct 1 14:17:37 2002 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 01 Oct 2002 09:17:37 -0400 Subject: [Python-Dev] Python-dev Summary for 2002-09-16 to 2002-09-30 References: Message-ID: <3D99A071.B7436B18@metaslash.com> Brett Cannon wrote: > > Ralf W. Grosse-Kunstleve brought up how a bug report about C++ > compatibility had not been handled yet. Both Tim and Guido said that if > he wanted a solution put in they would need a patch; neither of them are > C++ users. C++ uses should keep that in mind when trying to get C++ stuff > into the core. I've assigned the bug to me. Ralf provided a patch and I will checkin/backport soon. http://pyton.org/sf/607253 Neal From guido@python.org Tue Oct 1 14:38:47 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 01 Oct 2002 09:38:47 -0400 Subject: [Python-Dev] Dropping support for Tcl 8.0 and 8.1 In-Reply-To: Your message of "Tue, 01 Oct 2002 08:07:22 +0200." <200210010607.g9167Mns001471@mira.informatik.hu-berlin.de> References: <200210010607.g9167Mns001471@mira.informatik.hu-berlin.de> Message-ID: <200210011338.g91Dclo08817@odiug.zope.com> > In _tkinter.c, there is currently special code for Tcl 8.0 (no Tcl > objects) and Tcl 8.1 (no Unicode type). Tcl 8.2, 8.3, and 8.4 are > supported uniformly. > > Can we drop support for those two releases, in Python 2.3? If it's more work to keep support around than to drop it, sure. I agree there's probably nobody who's still using any Tcl version before 8.3 anyway; that's been out several years now. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Oct 1 14:50:18 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 01 Oct 2002 09:50:18 -0400 Subject: [Python-Dev] Python-dev Summary for 2002-09-16 to 2002-09-30 In-Reply-To: Your message of "Tue, 01 Oct 2002 13:48:44 +0200." <006d01c26940$7f3999e0$6d94fea9@newmexico> References: <006d01c26940$7f3999e0$6d94fea9@newmexico> Message-ID: <200210011350.g91DoIO08897@odiug.zope.com> > > ================================================= > > `builtins instance have modifiable __class__?`__ > > ================================================= > > __ http://mail.python.org/pipermail/python-dev/2002-September/028978.html > > > > Samuele Pedroni asked if being able to directly assigning to __class__ was > > allowed. Guido said "In 2.3, it's disallowed". It is highly discouraged > > in 2.2.2 but still doable. > > my question and the limitation refer only to builtin types (int,dict > ...), for instances of (user-defined) subtypes __class__ will be > still modifiable. Yes, setting __class__ for instances of user-defined new-style classes will be allowed (if the new __class__ value is compatible with the old). --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Tue Oct 1 16:17:05 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 01 Oct 2002 17:17:05 +0200 Subject: [Python-Dev] Dropping support for Tcl 8.0 and 8.1 In-Reply-To: <200210011338.g91Dclo08817@odiug.zope.com> References: <200210010607.g9167Mns001471@mira.informatik.hu-berlin.de> <200210011338.g91Dclo08817@odiug.zope.com> Message-ID: Guido van Rossum writes: > If it's more work to keep support around than to drop it, sure. I'm planning a number of changes which can be simplified if I don't have that many versions to support. Regards, Martin From guido@python.org Tue Oct 1 16:19:58 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 01 Oct 2002 11:19:58 -0400 Subject: [Python-Dev] Dropping support for Tcl 8.0 and 8.1 In-Reply-To: Your message of "Tue, 01 Oct 2002 17:17:05 +0200." References: <200210010607.g9167Mns001471@mira.informatik.hu-berlin.de> <200210011338.g91Dclo08817@odiug.zope.com> Message-ID: <200210011519.g91FJwI21585@odiug.zope.com> > > If it's more work to keep support around than to drop it, sure. > > I'm planning a number of changes which can be simplified if I don't > have that many versions to support. OK, +1. --Guido van Rossum (home page: http://www.python.org/~guido/) From gsw@agere.com Tue Oct 1 18:48:11 2002 From: gsw@agere.com (Gerald S. Williams) Date: Tue, 1 Oct 2002 13:48:11 -0400 Subject: [Python-Dev] Re: proposal for interfaces In-Reply-To: <20021001134101.18003.62032.Mailman@mail.python.org> Message-ID: I'm not sure why you'd need a new keyword for this feature when you can simply create a class that can check if methods in a derived class have been implemented. You can also check for other attributes if you use a live object instead of a class. All it needs to do is raise an exception if the interface isn't fully implemented. Here is an (admittedly simplified) example: import types from exceptions import TypeError class Interface(object): def implemented_by(self,implementer): # Verify against class methods: methods = [] for i in dir(self.__class__): if i not in dir(Interface): methods.append(i) for i in methods: if i not in dir(implementer): raise TypeError, "%s not implemented" % str(i) # and verify its type, parameters, ... # For objects, can also verify that attributes created in the # interface's __init__ exist: if type(implementer) not in (type(Interface), types.ClassType): # Assume it's an object. There's probably a better way. # (type(implementer) is types.ObjectType) works, but only # for new-style classes. methods += dir(Interface) attributes = [] for i in dir(self): if i not in methods: attributes.append(i) for i in attributes: if i not in dir(implementer): raise TypeError, "%s not implemented" % str(i) Something as simple as this would let you verify that a class or object has implemented an interface. For example: class TestInterface(Interface): def __init__(self): # object attributes to look for: self.attrib = "Some attribute" # class methods to look for: def method(self): "Some method" # To verify only methods, check the class TestInterface().implemented_by(Foo) # To verify attributes, check an object TestInterface().implemented_by(Foo()) I don't know what Zope does, but this doesn't seem that ugly to me. -Jerry From barry@python.org Tue Oct 1 19:18:27 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 1 Oct 2002 14:18:27 -0400 Subject: [Python-Dev] RELEASED email package version 2.4 Message-ID: <15769.59123.944370.58736@gargle.gargle.HOWL> I've just released the (standalone) email package version 2.4. This is identical to the version that is in the current cvs development tree for Python 2.3. It is also the version that I plan to back port to Python 2.2.2. I've also done a massive update to the web site documentation. Thanks to Fred Drake, the latter is actually in a usable format. :) email 2.4 is made available as a distutils package for installing in earlier versions of Python. It is compatible with Python 2.1.3, Python 2.2.1, and Python 2.3 cvs. email 2.4 is backward compatible with email version 1, which was released as part of Python 2.2.1. The email package is a library for parsing, handling, and generating email messages and other RFC 2822 style documents. It conforms to most of the email related RFCs including 2045-2047 (the MIME RFCs) and 2231. It is intended to replace several older modules in the standard distribution, such as rfc822, mimetools, multifile, mimify, and MIMEWriter, and such non-standard modules as mimecntl. email-2.4.tar.gz is available from the mimelib project on SourceForge: http://sf.net/projects/mimelib More information, as well as on-line documentation is available on the home page: http://mimelib.sf.net As this is the version I plan to back port to Python 2.2.2 in a few days, feedback, either on the code or the documentation, will be welcome. Enjoy, -Barry From pedronis@bluewin.ch Tue Oct 1 20:59:00 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 1 Oct 2002 21:59:00 +0200 Subject: [Python-Dev] perplexed by mro Message-ID: <01e101c26984$fc43da60$6d94fea9@newmexico> I'm trying to wrap my head around the mro computation in 2.2. First issue: PEP 253 and the descrintro document describe an algorithm, typeobject.c implements a rather different one. Descrintro mentions that. The algorithm described by descrintro, PEP253: "Using the classic lookup rule, construct the list of classes that would be searched, including duplicates. Now for each class that occurs in the list multiple times, remove all occurrences except for the last." which, if I understand correctly, means : do a left-right dfs of the class hierarchy and then discard all but the last occurence of each class. Now descrintro says that the result of this and the real algorithm in typeobject are the same, unless "two given base classes occur in a different order in the inheritance list of two different derived classes, and those derived classes are both inherited by yet another class" But here is an example where the criterium does not apply but still the algorithms' results diverge: >>> cpl.ex5(globals()) class a(object): pass class b(object): pass class c(object): pass class x(a): pass class y(a): pass class z(x,b,y,c): pass >>> cpl.names(z.__mro__) # just the class names, real 2.2 mro algo ['z', 'x', 'y', 'a', 'b', 'c', 'object'] >>> cpl.names(cpl.keeplast(cpl.lrdfs(z))) # descrintro simple algo ['z', 'x', 'b', 'y', 'a', 'c', 'object'] I was not able to find a new rule. My intuition is that when the result of the real mro has something to do with the dfs, it is sure from the beginning of it not the end, so the relation of the two algos is unclear. In general I have an hard time trying to predict the result of the real mro algo. Does it a have a list of constraints it enforces/preserves, of the qualities it has? I have tried to compare it to the algorithms used by Dylan and CLOS and others, described in this paper http://www.webcom.com/haahr/dylan/linearization-oopsla96.html they are described in terms such as quality criteria and constraints they preserve. Let's consider this example: >>> cpl.ex9(globals()) class a(object): pass class b(object): pass class c(object): pass class d(object): pass class e(object): pass class k1(a,b,c): pass class k2(d,b,e): pass class k3(d,a): pass class z(k1,k2,k3): pass >>> cpl.names(z.__mro__) ['z', 'k1', 'k3', 'a', 'k2', 'd', 'b', 'c', 'e', 'object'] the thing that leaves me most perplexed is that k3 appears before k2. All other algortihms I know would put k2 before k3. E.g. C3 (see paper) ... >>> cpl.names(cpl.c3_cpl(z)) ['z', 'k1', 'k2', 'k3', 'd', 'a', 'b', 'c', 'e', 'object'] or the descrintro simple algo: >>> cpl.names(cpl.keeplast(cpl.lrdfs(z))) ['z', 'k1', 'c', 'k2', 'b', 'e', 'k3', 'd', 'a', 'object'] This means that it does not preserve the local precedence lists (i.e. the inheritance list), it is not monotonic* because otherwise it would put d before a, it seems to preserve "some" aspects of the dfs traversal a,d vs d,a but still it puts k3 before k2. The other hypothesis, is that in this case, the algorithm should fail, because it cannot merge the result of mro(k1) merged with mro(k2) with mro(k3), because the latter is inconsistent with what obtained so far? >>> l=cpl.names(k1.__mro__) >>> l ['k1', 'a', 'b', 'c', 'object'] >>> r=cpl.names(k2.__mro__) >>> cpl.conservative_merge(l,r) ... >>> l ['k1', 'a', 'k2', 'd', 'b', 'c', 'e', 'object'] vs. >>> cpl.names(k3.__mro__) ['k3', 'd', 'a', 'object'] a Message-ID: <02a501c2699c$44718460$6d94fea9@newmexico> [me] > >>> cpl.ex5(globals()) > class a(object): pass > class b(object): pass > class c(object): pass > class x(a): pass > class y(a): pass > class z(x,b,y,c): pass > >>> cpl.names(z.__mro__) # just the class names, real 2.2 mro algo > ['z', 'x', 'y', 'a', 'b', 'c', 'object'] > >>> cpl.names(cpl.keeplast(cpl.lrdfs(z))) # descrintro simple algo > ['z', 'x', 'b', 'y', 'a', 'c', 'object'] > > I was not able to find a new rule. My intuition is that when the result of the > real mro has something to do with the dfs, it is sure from the beginning of it > not the end, so the relation of the two algos is unclear. to clarify this point: even in the absence of any kind of inconsistency, if a class INB (e.g. b above) get ordered as INB > R wrt to a class R recurring multiple times in the different superclasses' mros (e.g. a above) in between the first and last recurrence of that class R, the class INB will appear after R in the final order, while the simple algo (see also example) would put INB before R. In fact in the example: b follows a using the 2.2 mro; and b precedes a wrt the simple algo. Although in the example there's no kind of inconsistency. The mros of the superclasses of z being: x a object [mro x] b object [mro b] y a object [mro y] c object [mro c] The mro 2.2 algo merges these mros successively, also mro x is merged with mro b and the result is then merged with mro y and finally what has been obtained so far is merged with mro c. The merge step is as follows [python version of the c code in typeobject.c] def conservative_merge(left,right): while 1: left_size = len(left) right_size = len(right) eq = 0 for i in xrange(left_size): for j in xrange(right_size): if left[i] == right[j]: print i,j eq = 1 break if eq: break if eq: temp = [] for r in range(j): rr = right[r] if rr not in left: temp.append(rr) left[i:i] = temp right[0:j+1] = [] continue break left.extend(right) The algorithm move portions of right to left before merge points [shared elements], the rest of right is attached at the end of left. That's way the behavior explained above. The algorithm merges left and right as sorted sequences, favoring left in the case of a tie (assuming no incosistencies). Btw: the algorithms in the paper I have cited all use some kind of parallel merge of the superclasses' mros or some other constraints, not a sequential one as the 2.2 mro algo. regards. From pedronis@bluewin.ch Wed Oct 2 00:01:12 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Wed, 2 Oct 2002 01:01:12 +0200 Subject: [Python-Dev] perplexed by mro References: <01e101c26984$fc43da60$6d94fea9@newmexico> <02a501c2699c$44718460$6d94fea9@newmexico> Message-ID: <02d101c2699e$7067df40$6d94fea9@newmexico> > The algorithm move portions of right to left before merge points [shared > elements], the rest of right is attached at the end of left. That's way the > behavior explained above. s/way/why/ sorry. From arigo@ulb.ac.be Wed Oct 2 00:33:47 2002 From: arigo@ulb.ac.be (Armin Rigo) Date: Wed, 02 Oct 2002 01:33:47 +0200 Subject: [Python-Dev] Psyco requests and patches Message-ID: <3D9A30DB.FE2E042E@ulb.ac.be> Hello everybody, As I have been putting more work into Psyco to make sure it will work well with Python 2.3, I came up with a few support requests. Non-technical stuff first. People tend to think about Psyco as an obscure and complex piece of software for gurus -- should they know about it at all. Of course I would argue instead that it is exactly the piece of software that Python users should not have to worry about, because it should be perfectly transparent. As you can guess I would even say that the first widely-used dynamic programming language to natively include this kind of techniques would see a huge widening of its application domain. For Psyco to be a first step in that direction, here are a few goals that I would like you to comment. Publicity: well, I could do with a little more advertising, e.g. on www.python.org or better yet in the docs and tutorials, at places where performance is discussed. Right now I am not sure that you can learn about Psyco's existence unless you read python-dev or comp.lang.python. Ease of use: currently the psyco.bind() interface is easy to use, but it could be trivial instead of just easy. It should actually be fully transparent. (That's what psyco.jit() promised to do, but right now it badly sucks; keeping it around may have been an error, as people is actually trying it and getting bad results.) Portability: the easy part! Indeed, an empty module named 'psyco' portably provides exactly the same functionality as Psyco everywhere... On non-supported platforms you just get nothing better than the current interpreter performance, but nothing worse either. (Besides, emitting portable pseudo-code could still be a future option. Of course, emitting a platform's machine code is not that easy at all, but this fact doesn't compromise portability.) Ease of maintenance: the hard part. Psyco is not currently ready for that. The implementation is way too obscure. You have no guarantee if I tell -- and I do -- that I plan to upgrade Psyco to all newer Python versions for at least as long as the 2.x series is alive. We need more than one person involved. (Note that upgrades are not as time-consuming as, say, in the case of Christian Tismer's first Stackless implementation that required a complete rewrite of the interpreter.) Let's defer this point to the bottom of this e-mail. Inclusion in the Standard Library or the interpreter core itself: not right now! This is a guideline for the goals discussed above. I might be inclined to think that completion of all the previous goals would be enough for consideration...? -=- Python patches -=- Psyco requires no change in the interpreter core -- however, there are a couple of things that needed obfuscated workarounds or just don't completely work at all. A few minor changes would make it all much cleaner. I submitted three patches to SF (http://www.python.org/sf/617309, http://www.python.org/sf/617311, http://www.python.org/sf/617312). The patches should be clear enough, not introducing obscure "features" or measurable performance loss. The diffs are currently against the 2.2 branch -- it would greatly help Psyco if the patches made their way into Python 2.2.2, which I understood to be the candidate Python-in-a-tie. I will submit 2.3 versions of the patches as well, of course. -=- Maintenance of Psyco -=- Psyco is currently highly obscure. I want to rewrite it in Python. Still, however, there will always be the problem that for any feature supported by Psyco someone must track the changes in the corresponding feature in the interpreter, and port them. (This doesn't concern the features not supported by Psyco: for these it always calls their up-to-date C implementation.) This process might be semi-automated with a tool that would be able to dig into C code and analyse it, but this would be sort of a hack. We might do some thinking about this synchronization problem when the discussion about Python 3 is raised again: how hard would it be e.g. to write some parts of the interpreter directly in Python -- or at least in some specific dialect that can be turned into C (a la Pyrex) for ease of compilation? It may actually help make the interpreter more understandable. This looks a bit far-fetched, but I will never give up my claims that high-level languages are just better than low-level ones for most purposes, and writing an interpreter is no exception! For example, there was a December 2000 proposal to rewrite the "cmath" module in Python (http://www.python.org/sf/210838). It never actually made its way into the Standard library, which is too bad for Psyco, because Psyco-with-cmath-in-Python would be faster than Psyco-with-cmath-in-C. Practically, the whole point of having a single high-level source code is that you can do more than just one thing with it. With C code, it is a horrible hack to do anything else than compile and execute it. With Python code it's a different matter. You can use this single source to produce both the equivalent C code and to produce Psyco itself. In each case there is still an important part written directly in C, of course; I am not (yet;-) suggesting that Python should be written *only* in Python! Very roughly, the separation would be between the built-in objects + abstract protocols in 'Objects/' (which could be written in Python) and the core in 'Python/' (which could not). The framework required to do such a thing looks complicated. I would argue that it probably isn't (and in any case it might be worth being carefully considered); this extra complexity might be favorably balanced if only just because writing Python in Python is easier than in C :-) A bientot, Armin PS: I just obtained my math Ph.D. -- it's now Psyco time again! From tismer@tismer.com Wed Oct 2 00:42:14 2002 From: tismer@tismer.com (Christian Tismer) Date: Tue, 01 Oct 2002 16:42:14 -0700 Subject: [Python-Dev] Psyco requests and patches References: <3D9A30DB.FE2E042E@ulb.ac.be> Message-ID: <3D9A32D6.8070405@tismer.com> Armin Rigo wrote: ... > PS: I just obtained my math Ph.D. -- it's now Psyco time again! Congratulations! -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer@tismer.com Wed Oct 2 00:59:56 2002 From: tismer@tismer.com (Christian Tismer) Date: Tue, 01 Oct 2002 16:59:56 -0700 Subject: [Python-Dev] regex Perl vs Python Message-ID: <3D9A36FC.304@tismer.com> Hi Friends, It is said that with the new re module, Python "has the regular expressions like Perl". Can somebody tell me to what extent this is true? I need to convince s.o. not to use Perl for a larger project, but I cannot answer the question up to which version the Python re's are covering all of Perl's? thanks a lot - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From martin@v.loewis.de Wed Oct 2 07:41:59 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 02 Oct 2002 08:41:59 +0200 Subject: [Python-Dev] Psyco requests and patches In-Reply-To: <3D9A30DB.FE2E042E@ulb.ac.be> References: <3D9A30DB.FE2E042E@ulb.ac.be> Message-ID: Armin Rigo writes: > The diffs are currently against the 2.2 branch -- it would greatly > help Psyco if the patches made their way into Python 2.2.2 That can't be done: they break binary compatibility (atleast 617309 does, haven't checked the others), which is unacceptable for a micro release. Regards, Martin From jrw@pobox.com Wed Oct 2 08:10:22 2002 From: jrw@pobox.com (John Williams) Date: Wed, 02 Oct 2002 02:10:22 -0500 Subject: [Python-Dev] *very* revised proposal for interfaces Message-ID: <3D9A9BDE.7040304@pobox.com> After re-reading PEPs 245 (interface syntax) and 246 (object adaptation), I hit on a new idea that seems to do everything these two PEPs and my original proposal (plus revisions) do, all with much less fuss. I also realized I was adding a lot of excess complexity in order to implement all the rigorous error checking that doesn't really make sense in the Python world anyway (thanks for nudging me in that direction!) As a side effect it's also a lot easier to explain. Here's the new version, in a no-frills stripped-down form, explained by example. # Create an interface--not much new here, just simpler syntax and fewer frills. class I(interface): prop1 = "Docstring for property prop1." # It would be good form to put an argument list here, # but this method will never be called so it's not necessary. def meth1(): "Docstring for method meth1." # Create a class implementing the interfac. # It inherits from "object" explicitly, since a class whose bases # are all interfaces is itself an interface, not an implementation. class C(object, I): def meth(self,a,b): print "This is I.meth1 implemented by C." # (I'm not quite sure how to handle properies yet.) assert I not in C.__bases__ # Interfaces are not really base classes! # Example method requiring an object supporting interface I. def usingI(x,y): # Check that x supports interface I. Return either x or a suitable # wrapper around x that supports I's methods. If x doesn't support I, # raise an exception. x = I(x) # Check that y also supports I. If it doesn't, return None instead of # raising an exception. y = I(y, None) ... # Example of binding an existing class to an interface. Like creating # an ordinary implementation, the bases are an interface and a regular # class, but instead of creating a usable new class, it just associates # an interface with a class and defines how the interface's methods are # implmented. The magic class name __binding__ triggers this behavior. # Instances of this wrapper class are created by calling interface I. class __binding__(str, I): def meth1(self,a,b): # Say what meth1 does on strings. assert isinstance(self, str) print "I.meth1 called on", self # Example of using __binding__ to resolve a conflict between two interfaces. class J(interface): def meth1(): "No relation to I.meth1!" class D(object, I, J): # Implement two interfaces. # Define *both* I.meth1 and J.meth1 for this class. def meth1(self,a,b): print "This is I.meth1 implemented by D." class __binding__(D,J): def meth1(self): print "This is J.meth1 implemented by D." # Now to call these two methods, we can do this: d = D() i = I(d) # Note i could really just be d here, not that it matters. i.meth1(1,2) # Prints "This is I.meth1 implemented by D." j = J(d) # Here j must be a wrapper to make things work right. j.meth1() # Prints "This is J.meth1 implemented by D." # Or use different syntax to be more explicit. I.meth1(d,1,2) # Same as I(d).meth1(1,2) J.meth1(d) # Same as J(d).meth1() Please let me know if I'm just insane for sleep deprivation! jw From arigo@tunes.org Wed Oct 2 11:54:50 2002 From: arigo@tunes.org (Armin Rigo) Date: Wed, 2 Oct 2002 12:54:50 +0200 (CEST) Subject: [Python-Dev] Psyco requests and patches In-Reply-To: Message-ID: Hello Martin, On 2 Oct 2002, Martin v. Loewis wrote: > (about Psyco-friendly patches for 2.2.2) > That can't be done: they break binary compatibility (atleast 617309 > does, haven't checked the others), which is unacceptable for a micro > release. No, I don't think any of the patches breaks binary compatibility. I have kept this in mind while making them. Two patches add an extra field each to the PyThreadState structure, but at the end. 617309: the new hook is of no concern to extension modules. Reading the 'frame' field is still allowed. The only real change is into the implementation of PyEval_GetFrame(). A bientot, Armin. From gsw@agere.com Wed Oct 2 13:59:03 2002 From: gsw@agere.com (Gerald S. Williams) Date: Wed, 2 Oct 2002 08:59:03 -0400 Subject: [Python-Dev] Re: *very* revised proposal for interfaces In-Reply-To: <20021002105618.14812.46853.Mailman@mail.python.org> Message-ID: John Williams wrote: > Here's the new version, in a no-frills stripped-down form, explained by > example. ... > class I(interface): > prop1 = "Docstring for property prop1." > # It would be good form to put an argument list here, > # but this method will never be called so it's not necessary. > def meth1(): "Docstring for method meth1." This is along the lines of what I was suggesting. Note that you can optionally do some argument list checking. You can also check other properties but that requires you to check against an instantiated object and indicate the presence of these properties when instantiating an interface object. > class C(object, I): ... > assert I not in C.__bases__ # Interfaces are not really base classes! Isn't this assertion going to fail the way you showed it? Inheritance needn't be used at all for interfaces. > # Example of binding an existing class to an interface. Like creating > # an ordinary implementation, the bases are an interface and a regular > # class, but instead of creating a usable new class, it just associates > # an interface with a class and defines how the interface's methods are > # implmented. The magic class name __binding__ triggers this behavior. > # Instances of this wrapper class are created by calling interface I. I don't think you need any magic class names. If a class wants to present an interface, it can provide a derived class and/or surrogate if needed. For example: class MyDerivedInterface(MyClass): def interface_name(self,x): return MyClass.my_name(self,x,1) class MySurrogateInterface(object): def __init__(self,me): self.me = me def __getattr__(self,attr): return self.me.__getattribute__(attr) def interface_name(self,x): return self.me.my_name(x,1) Either one implements the interface. > d = D() > i = I(d) # Note i could really just be d here, not that it matters. But do interfaces really need to have this capability built-in? If you want to present an interface that's constrained, you can do it using a derived class or surrogate. I would view "d" as the correct thing to use in this example. ----- One thing my simplified example did not show (though it was in my local versions) is the ability to have implemented_by() register the interface in addition to testing compliance. This requires something like "implementerClass.__implements__ += interfaceClass" (with the appropriate safety checks). This would allow a more "contractual" approach to interfaces, since you could check to see if a class has been registered as supporting an interface before trying to use that interface. ----- I think we're both saying the same thing--there is a core issue that can be addressed without language extensions or similar things that complicate the implementation. But I have a more minimalist view: don't impact anything if your class presents the interface already. All you need is a simple means to declare (and verify) that a class implements an interface. -Jerry From jrw@pobox.com Wed Oct 2 20:55:58 2002 From: jrw@pobox.com (John Williams) Date: Wed, 02 Oct 2002 14:55:58 -0500 Subject: [Python-Dev] re: *very* revised proposal for interfaces Message-ID: <3D9B4F4E.8080208@pobox.com> Gerald S. Williams wrote: > This is along the lines of what I was suggesting. Note that > you can optionally do some argument list checking. You can > also check other properties but that requires you to check > against an instantiated object and indicate the presence of > these properties when instantiating an interface object. I think we're trying to solve different problems. You seem to want to very that a class supports a certain set of methods, but I just want to verify that the class *claims* to implement the interface and provide a framework to make that happen, like a more automated version of PEP 245. For my purposes, I'm not sure the error checking is really a Good Thing after some of the things Esteban Castro pointed out. If you fail to implement the nesessary methods, that's your own problem. This approach is a lot less work for me, and handles strange cases of late binding more gracefully. Of course a warning would be helpful in the 95% of cases where the assumptions made for error checking are correct. >>class C(object, I): > > ... > >>assert I not in C.__bases__ # Interfaces are not really base classes! > > Isn't this assertion going to fail the way you showed it? > Inheritance needn't be used at all for interfaces. I use inheritence syntax for interfaces, since it's there and convenient, but I wanted to emphasize that although interfaces can be subclasses of interfaces, regular classes *cannot* be subclasses of interfaces, even though the syntax makes it look that way. It might be worthwhile to add an "implements(class, interface)" function. If you want to get fancy, let the first argument be an instance as well, and let the second argument be a list of interfaces, all of which must be supported (the opposite of isinstance's disjunctive behavior!) > I don't think you need any magic class names. If a class > wants to present an interface, it can provide a derived > class and/or surrogate if needed. For example: > > class MyDerivedInterface(MyClass): > def interface_name(self,x): return MyClass.my_name(self,x,1) > > class MySurrogateInterface(object): > def __init__(self,me): self.me = me > def __getattr__(self,attr): return self.me.__getattribute__(attr) > def interface_name(self,x): return self.me.my_name(x,1) This only works when you can add methods to classes that implement the interface. It's a bit awkward for retrofitting classes you didn't write, and just not possible for classes implmented in C, like the builtin classes. >>d = D() >>i = I(d) # Note i could really just be d here, not that it matters. > > But do interfaces really need to have this capability > built-in? If you want to present an interface that's > constrained, you can do it using a derived class or > surrogate. I would view "d" as the correct thing to > use in this example. The idea of constraining i to only implement methods of I is something from my original proposal that I've since decided was a bad idea--extra work for no benefit. The real issue here is that I(d) implements the methods of interface I. If class D implements the methods of I directly, then I(d) can be d, but in normal use you won't know the class of d, and a wrapper may be necessary, so you can't assume I(d) is d. > One thing my simplified example did not show (though > it was in my local versions) is the ability to have > implemented_by() register the interface in addition > to testing compliance. This requires something like > "implementerClass.__implements__ += interfaceClass" > (with the appropriate safety checks). This would > allow a more "contractual" approach to interfaces, > since you could check to see if a class has been > registered as supporting an interface before trying > to use that interface. This is more like what I had in mind to do behind the scenes. > I think we're both saying the same thing--there is > a core issue that can be addressed without language > extensions or similar things that complicate the > implementation. But I have a more minimalist view: > don't impact anything if your class presents the > interface already. All you need is a simple means > to declare (and verify) that a class implements an > interface. I think we're more in agreement than you think. My original proposal doesn't do what you're saying here, but the revised version with slightly relaxed rules does, by allowing an object to be its own wrapper/proxy when it implements the interface in the most straightforward way, so the "wrapping" step (calling the interface class) just reduces to a type assertion. jw From mchermside@ingdirect.com Wed Oct 2 21:27:31 2002 From: mchermside@ingdirect.com (Chermside, Michael) Date: Wed, 2 Oct 2002 16:27:31 -0400 Subject: [Python-Dev] *very* revised proposal for interfaces Message-ID: <902A1E710FEAB740966EC991C3A38A8903C35AE8@INGDEXCHANGEC1.ingdirect.com> [Please excuse poor email format... I've just been moved=20 to outlook and haven't yet figured out how to make it behave like a normal email program.] John, I really like your proposal!!! The main thing that I like about it is that, as you say: "It's also a lot easier to explain" That's usually a sign of a good design... when it all falls into place and becomes simple. I will admit, I've been trying to follow the various interface suggestions and have mostly been getting confused. This is simple and elegant, even if a few bits would improve with tweaking. Here is a running commentary on things I liked and disliked: 1. You create a new built-in type called "interface" with the special property that it does NOT appear in __bases__ even if declared to do so. I like this. Your syntax for the most common case of=20 supporting an interface looks like this: class C(object, I): <... code here ...> If we DIDN'T create a "special" type (special because Python has to omit pure interfaces in generating __bases__), then things could still work by some sort of declaration after C is created: class C(object): <... code here ...> declareItImplements(C,I) But that's putting it in the wrong place. The declaration to support an interface should be easy to type and to read (in the=20 common case), and should come at the TOP of the declaration of the class, not the end. So I like this. 2. You have a "magic class" __binding__. This sounds odd. Python hasn't ever used a "magic class" before. But it certainly makes the syntax simple. After all, you need to define a bunch of methods (the methods needed to conform to the interface). Declaring a class is the handy way to define a bunch of methods. So I think this is a little weird, but I don't really object. If there's another solution, though, I'd like to hear it. 3. Where does the "binding" live? Creating some sort of "global binding registry" is really not a good idea. Global is just bad, it becomes hard to find where things were set, which causes mysterious code failures, and so forth. At first, that's what I thought you were doing, but a closer look changed my mind. It seems to me that when you create a magic "__binding__" class what you are ACTUALLY doing is adding to a list of "classes implementing it" which is maintained in and associated with I,=20 the interface object. That has ABSOLUTELY NO effect on anyone who is not using the interface, and it feels like the "right" place to store this information. So I if I understand this correctly, I like it. 4. You make use of the interface by invoking it on an object: i =3D I(d) This has been discussed before, and it's a good approach. It allows for interfaces to return the object (thus not creating extra objects unnecessarily) or a proxy (thus giving lots of extra flexibility). Solidly supported. 5. You do NOT provide a way to do type declarations along with the interface. Python isn't picky about types, it naturally supports a policy=20 of allowing you to use any object that will do the job. That's a good thing for the language (even if my own preferences tend=20 toward a stricter typing system) and interfaces aren't about undermining it... just strengthening it! So +1 here. 6. You do NOT provide a way to declare preconditions, postconditions, or anything like this for the methods. In fact, there's no specification other than a docstring. For this I'm not sure. I like the minimalistic, simple approach of just giving a docstring, but I might also like some of the power of extra features. If I have to give up extra power to get a simple, workable approach then I'll live with it. 7. There's no way to "extend" interfaces, since inheriting from an interface means something different. That's OK with me -- seems like something one would only do when in a severe meta-programming mood. So all in all, I'm liking what I'm seeing here. -- Michael Chermside =20 From tismer@tismer.com Wed Oct 2 22:05:49 2002 From: tismer@tismer.com (Christian Tismer) Date: Wed, 02 Oct 2002 14:05:49 -0700 Subject: [Python-Dev] Urgent: regex Perl vs Python References: <3D9A36FC.304@tismer.com> Message-ID: <3D9B5FAD.4020300@tismer.com> Christian Tismer wrote: > Hi Friends, > > It is said that with the new re module, Python "has the regular > expressions like Perl". > Can somebody tell me to what extent this is true? > I need to convince s.o. not to use Perl for a larger project, > but I cannot answer the question up to which version the > Python re's are covering all of Perl's? Please, can somebody give me just a version number? I really need this info, and I'm not intending to start a thread, I sware. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From pyth@devel.trillke.net Wed Oct 2 22:14:55 2002 From: pyth@devel.trillke.net (holger krekel) Date: Wed, 2 Oct 2002 23:14:55 +0200 Subject: [Python-Dev] Urgent: regex Perl vs Python In-Reply-To: <3D9B5FAD.4020300@tismer.com>; from tismer@tismer.com on Wed, Oct 02, 2002 at 02:05:49PM -0700 References: <3D9A36FC.304@tismer.com> <3D9B5FAD.4020300@tismer.com> Message-ID: <20021002231455.C30315@prim.han.de> Christian Tismer wrote: > Christian Tismer wrote: > > Hi Friends, > > > > It is said that with the new re module, Python "has the regular > > expressions like Perl". > > Can somebody tell me to what extent this is true? > > I need to convince s.o. not to use Perl for a larger project, > > but I cannot answer the question up to which version the > > Python re's are covering all of Perl's? > > Please, can somebody give me just a version number? > I really need this info, and I'm not intending > to start a thread, I sware. hi chris, Maybe you could mail the current regex maintainer (IIRC Frederik Lundh). Or work yourself and dig through google :-) ciao holger From gsw@agere.com Wed Oct 2 22:31:08 2002 From: gsw@agere.com (Gerald S. Williams) Date: Wed, 2 Oct 2002 17:31:08 -0400 Subject: [Python-Dev] Re: *very* revised proposal for interfaces In-Reply-To: <3D9B4DCB.6040206@pobox.com> Message-ID: John Williams wrote: > I think we're trying to solve different problems. You seem to want to > very that a class supports a certain set of methods, but I just want to > verify that the class *claims* to implement the interface and provide a > framework to make that happen, like a more automated version of PEP 245. They're aspects of the same problem. When the claim is made, it should be verified to some degree in case the interface changes out from beneath the implementer (to what degree is certainly negotiable, though). Checking for late binding and generally anything other than the presence of unbound class methods requires an object, although it can still be done. Or you could have ways to explicitly mark these as supported (perhaps a parameter to the implemented_by method in my example). > This only works when you can add methods to classes that implement the > interface. It's a bit awkward for retrofitting classes you didn't > write, and just not possible for classes implmented in C, like the > builtin classes. No matter what you have under the hood, you should at least be able to wrap it entirely. My point is that if you want to provide a new interface, just do it with existing mechanisms. (If there are cases where this isn't possible, adding a way to define class-type interfaces that cannot be expressed now seems wrong to me.) > > I think we're both saying the same thing [ ... ] > I think we're more in agreement than you think. Yes, we're in violent agreement. :-) Well, at least we're converging... Actually, there is still one sticking point: the presence of wrappers and proxys. IMO, mechanisms exist to implement these if needed, so I'd rather leave them out (and consequently not have to change the Python language at all). -Jerry From guido@python.org Wed Oct 2 22:34:00 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 02 Oct 2002 17:34:00 -0400 Subject: [Python-Dev] Urgent: regex Perl vs Python In-Reply-To: Your message of "Wed, 02 Oct 2002 14:05:49 PDT." <3D9B5FAD.4020300@tismer.com> References: <3D9A36FC.304@tismer.com> <3D9B5FAD.4020300@tismer.com> Message-ID: <200210022134.g92LY0Q05792@pcp02138704pcs.reston01.va.comcast.net> > > It is said that with the new re module, Python "has the regular > > expressions like Perl". > > Can somebody tell me to what extent this is true? > > I need to convince s.o. not to use Perl for a larger project, > > but I cannot answer the question up to which version the > > Python re's are covering all of Perl's? > > Please, can somebody give me just a version number? > I really need this info, and I'm not intending > to start a thread, I sware. Nobody knows, Chris. We started with Perl regexes as they were in Perl version 5.x (for some low value of x that nobody remembers), threw out everything that Python couldn't implement (like references to variables with $name), threw away a few things that we felt were only present for Perl 4 backwards compatibility (like some of the ambiguity between octal character references and backreferences), and added some Python specific features like named groups. Hardly any new Perl regex features were added, but I can't guarantee that we didn't add anything. Your best bet is to compare the reference docs for Python and Perl regexes. --Guido van Rossum (home page: http://www.python.org/~guido/) From python-pep@ccraig.org Wed Oct 2 23:11:41 2002 From: python-pep@ccraig.org (python-pep@ccraig.org) Date: 02 Oct 2002 18:11:41 -0400 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues Message-ID: I just uploaded a reference implementation of how rationals might look in Python as patch 617779 [1]. I do have some new issues for discussion that I'd like to get some comments on before I change the PEP. 1) Should future division return rationals rather than floats. I had sort of assumed this would happen, but I just had a discussion with Kirby Urner and couldn't convince him it was a good idea, so I guess it isn't so clear. Arguments for: - you don't lose precision on divides - It provides a really nice way to specify rationals (i.e. 1/3) - It allows you to eventually unify int/long/rationals so that rationals with a denominator of 1 are automagically upcast. Arguments against: - people who have already changed their code to expect floats will have to change it again - rationals are slow 2) Should floats compare equal with rationals only when they are equal, or whenever the are the closest float? (i.e. will .2 compare equal to rational(1, 5)) 3) Should rationals try to hash the same as floats? My leaning on this is that it will be decided by (2). If they compare equal when 'close enough' then they should hash the same, if not then they should only hash the same when both are integral. I would rather not see .5 hash with rational(1, 2) but not .2 with rational(1, 5). [1] http://sourceforge.net/tracker/?func=detail&aid=617779&group_id=5470&atid=305470 -- Christopher A. Craig "[Windows NT is] not about 'Where do you want to go today?'; it's more like 'Where am I allowed to go today?'" -- Mike Mitchell, NT Systems Administrator From esr@thyrsus.com Wed Oct 2 23:14:21 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Wed, 2 Oct 2002 18:14:21 -0400 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: References: Message-ID: <20021002221421.GA29958@thyrsus.com> python-pep@ccraig.org : > I just uploaded a reference implementation of how rationals might look > in Python as patch 617779 [1]. I do have some new issues for > discussion that I'd like to get some comments on before I change the > PEP. > > 1) Should future division return rationals rather than floats. I had > sort of assumed this would happen, but I just had a discussion with > Kirby Urner and couldn't convince him it was a good idea, so I guess > it isn't so clear. > > Arguments for: > - you don't lose precision on divides > - It provides a really nice way to specify rationals (i.e. 1/3) > - It allows you to eventually unify int/long/rationals so that > rationals with a denominator of 1 are automagically upcast. > > Arguments against: > - people who have already changed their code to expect floats will > have to change it again > - rationals are slow +1 for returning rationals. It's the right thing -- and if it fails, it will fail noisily, right? > 2) Should floats compare equal with rationals only when they are > equal, or whenever the are the closest float? (i.e. will .2 > compare equal to rational(1, 5)) > > 3) Should rationals try to hash the same as floats? My leaning on > this is that it will be decided by (2). If they compare equal when > 'close enough' then they should hash the same, if not then they should > only hash the same when both are integral. I would rather not see .5 > hash with rational(1, 2) but not .2 with rational(1, 5). APL faced this problem twenty-five years ago. I like its solution; a `fuzz' variable defining the close-enough-for-equality range. -- Eric S. Raymond From ark@research.att.com Wed Oct 2 23:27:22 2002 From: ark@research.att.com (Andrew Koenig) Date: 02 Oct 2002 18:27:22 -0400 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021002221421.GA29958@thyrsus.com> References: <20021002221421.GA29958@thyrsus.com> Message-ID: Eric> APL faced this problem twenty-five years ago. I like its Eric> solution; a `fuzz' variable defining the Eric> close-enough-for-equality range. I used to like APL's approach, but I've changed my mind. Part of the reason is that there are some places where unfuzzed comparison is essential, such as sorting. Another part is that fuzzy comparison destroys substitutability: If a==b, it is not always true that f(a)==f(b). Much as I like APL, I'd rather use Scheme's numeric model. -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From esr@thyrsus.com Wed Oct 2 23:30:07 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Wed, 2 Oct 2002 18:30:07 -0400 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: References: <20021002221421.GA29958@thyrsus.com> Message-ID: <20021002223007.GA30048@thyrsus.com> Andrew Koenig : > Eric> APL faced this problem twenty-five years ago. I like its > Eric> solution; a `fuzz' variable defining the > Eric> close-enough-for-equality range. > > I used to like APL's approach, but I've changed my mind. > > Part of the reason is that there are some places where unfuzzed > comparison is essential, such as sorting. Another part is that > fuzzy comparison destroys substitutability: If a==b, it is > not always true that f(a)==f(b). > > Much as I like APL, I'd rather use Scheme's numeric model. Good points...but the fuzz variable could default to zero. It didn't in APL, which I always thought a mistake. -- Eric S. Raymond From jrw@pobox.com Wed Oct 2 23:36:25 2002 From: jrw@pobox.com (John Williams) Date: Wed, 02 Oct 2002 17:36:25 -0500 Subject: [Python-Dev] *very* revised proposal for interfaces References: <902A1E710FEAB740966EC991C3A38A8903C35AE8@INGDEXCHANGEC1.ingdirect.com> Message-ID: <3D9B74E9.5010704@pobox.com> Chermside, Michael wrote: > John, I really like your proposal!!! Thanks! > 2. You have a "magic class" __binding__....So I think this is a > little weird, but I don't really object...If there's another > solution, though, I'd like to hear it. That's pretty much my feeling, too. This is an area that could probably benefit from adding some language support, since I'm already pushing the class syntax pretty far. The idea for the magic class names came from the observation that class/interface bindings really are classes in themselves, but there's no need to ever refer to them directly, so they don't need real names. > 3. Where does the "binding" live? > > Creating some sort of "global binding registry" is really not > a good idea. Global is just bad, it becomes hard to find where > things were set, which causes mysterious code failures, and so > forth. At first, that's what I thought you were doing, but a > closer look changed my mind. > > It seems to me that when you create a magic "__binding__" class > what you are ACTUALLY doing is adding to a list of "classes > implementing it" which is maintained in and associated with I, > the interface object. That has ABSOLUTELY NO effect on anyone > who is not using the interface, and it feels like the "right" > place to store this information. So I if I understand this > correctly, I like it. Yes. It was pointed out to me privately that you can still make a mess if you have multiple bindings for the same class/interface pair (perhaps in a large project). My best answer to that concern so far is to say that (as a matter of style) bindings should always live with either the interface of the class they bind, and to enforce a rule that there may be only one explictit __binding__ per class/interface pair. > 6. You do NOT provide a way to declare preconditions, postconditions, > or anything like this for the methods. In fact, there's no > specification other than a docstring. > > For this I'm not sure. I like the minimalistic, simple approach > of just giving a docstring, but I might also like some of the > power of extra features. If I have to give up extra power to get > a simple, workable approach then I'll live with it. I'd really like to put more extensible error checking in, since it was in my original wish list. I left it out here because the proposal looks complete without it, and no solution I've come up with yet seems really satisfactory. I would welcome suggestions on how to do this well. As for syntax, I'm currently learning towards using magic methods named __pre__foo and __post__foo to check a method named foo. I'm kind of uneasy about extending special treatment from specific identifiers to *all* identifiers with a given prefix, but I suppose there is a precendent in the special treatment of module-level identifiers ("_" == no export) and class-level identifiers ("__" == pseudo-private). It's also fairly concise and expressive. > 7. There's no way to "extend" interfaces, since inheriting from an > interface means something different. > > That's OK with me -- seems like something one would only do when > in a severe meta-programming mood. Actually you can create derived interfaces, if that's what you meant. If all of a class's bases are interfaces, the new class is an interface as well. I've seen subclassed interfaces put to very good use in the C++ STL, although the interfaces are just a documentation convention there. I'd be surprised if the Java standard libraries didn't contain a lot of dervied interfaces--I know the language allows it, even if it's not often used. Thanks (to everyone) for all the commentary. I feel like I'm making a lot of progress towards something that's functional and pleasant to use, but there are still a lot of open issues. I'll post a minimal reference implementation when I get done with it and hopefully generate some more interest in the proposal. I expect that when I start putting in more features there will be more cases where no solution is obviously the right one, so I'll need all the input I can get to avoid doing things in ways that seem backwards to 90% of my target audience. jw From pinard@iro.umontreal.ca Wed Oct 2 23:56:49 2002 From: pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois?= Pinard) Date: Wed, 02 Oct 2002 18:56:49 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021002221421.GA29958@thyrsus.com> ("Eric S. Raymond"'s message of "Wed, 2 Oct 2002 18:14:21 -0400") References: <20021002221421.GA29958@thyrsus.com> Message-ID: [Eric S. Raymond] >> 1) Should future division return rationals rather than floats. > +1 for returning rationals. It's the right thing -- and if it fails, > it will fail noisily, right? While I agree with the theoretical arguments, I have the practical fear that rationals could grow very big, rather quickly, in the course of a long computation involving them in various ways. By big, I mean the numerator and denominator of the fraction taken in isolation, not the number itself. Consider inversions of an integer matrices, approximations with truncated series, or worse things like, maybe, discrete Fourier transforms. Bigger rationals are, slower they become, and more memory they take. The danger is that programmers may get surprised or hurt by Python performance degradation, raising frequent and recurrent questions here and elsewhere. On the other hand, I would love if Python was not loosing precision on non-truncating integer division, so let me try a bit to destroy my own fears. On average, most programs will not use matrices of rational numbers, nor play with series. Moreover, most programs do not use so many different numeric variables anyway, nor perform long computations involving them. Many programs do not go beyond adding or subtracting one, once in a while! So, I would guess that _on the average_, using rational numbers might be acceptable and go almost unnoticed by most people. So it might be more worth accepting as a community to warn programmers who are more prone to numerical algorithms of the intrinsic dangers of integer division in Python. But those feelings are no proof of anything. How do we get the confirmation that using rationals in Python would be easy going and innocuous in practice, beforehand? It would surely be nice relying in such a feature! -- François Pinard http://www.iro.umontreal.ca/~pinard From tismer@tismer.com Thu Oct 3 00:02:05 2002 From: tismer@tismer.com (Christian Tismer) Date: Wed, 02 Oct 2002 16:02:05 -0700 Subject: [Python-Dev] Urgent: regex Perl vs Python References: <3D9A36FC.304@tismer.com> <3D9B5FAD.4020300@tismer.com> <200210022134.g92LY0Q05792@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3D9B7AED.1040004@tismer.com> [me about compatibility to which Perl version] Holger: > Maybe you could mail the current regex maintainer (IIRC Frederik Lundh). > > Or work yourself and dig through google :-) I'm at a customer in the US and cannot spend hours on this (and re stuff is not my virtue), so I tried to ask for help. Guido: > Nobody knows, Chris. We started with Perl regexes as they were in > Perl version 5.x (for some low value of x that nobody remembers), > threw out everything that Python couldn't implement (like references > to variables with $name), threw away a few things that we felt were > only present for Perl 4 backwards compatibility (like some of the > ambiguity between octal character references and backreferences), and > added some Python specific features like named groups. Hardly any new > Perl regex features were added, but I can't guarantee that we didn't > add anything. Ok, that's a lot of help to me. > Your best bet is to compare the reference docs for Python and Perl > regexes. Will try to defer this to the client, after giving him the the above info. Many thanks - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer@tismer.com Thu Oct 3 00:13:57 2002 From: tismer@tismer.com (Christian Tismer) Date: Wed, 02 Oct 2002 16:13:57 -0700 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <20021002221421.GA29958@thyrsus.com> Message-ID: <3D9B7DB5.4000702@tismer.com> Fran=E7ois Pinard wrote: > [Eric S. Raymond] >=20 >=20 >>>1) Should future division return rationals rather than floats. >> >=20 >>+1 for returning rationals. It's the right thing -- and if it fails,=20 >>it will fail noisily, right? +1 here, too! > While I agree with the theoretical arguments, I have the practical fear= that > rationals could grow very big, rather quickly, in the course of a long > computation involving them in various ways. By big, I mean the numerat= or > and denominator of the fraction taken in isolation, not the number itse= lf. > Consider inversions of an integer matrices, approximations with truncat= ed > series, or worse things like, maybe, discrete Fourier transforms. Yes, this fear is right. I think this is just great. Let it grow! Let the user feel what precision he's carrying around, and how much they throw away when they reduce down to float. No I think this is really of advantage. Exact is better than small, and it is all in the user's hand. Make a float() and you're done. > Bigger rationals are, slower they become, and more memory they take. T= he > danger is that programmers may get surprised or hurt by Python performa= nce > degradation, raising frequent and recurrent questions here and elsewher= e. >=20 > On the other hand, I would love if Python was not loosing precision on > non-truncating integer division, so let me try a bit to destroy my own > fears. On average, most programs will not use matrices of rational num= bers, > nor play with series. Moreover, most programs do not use so many diffe= rent > numeric variables anyway, nor perform long computations involving them. > Many programs do not go beyond adding or subtracting one, once in a whi= le! I don't think one was considering to go rationale, all the time, just as the result of integer divide? To me, rationales would be the natural superclass of integers. Floats would stand somewhere else, really different animals. > So, I would guess that _on the average_, using rational numbers might b= e > acceptable and go almost unnoticed by most people. So it might be more > worth accepting as a community to warn programmers who are more prone t= o > numerical algorithms of the intrinsic dangers of integer division in Py= thon. >=20 > But those feelings are no proof of anything. How do we get the confirm= ation > that using rationals in Python would be easy going and innocuous in > practice, beforehand? It would surely be nice relying in such a featur= e! I'm all for it. Get correct results in the first place. Cut precision explicitly when needed, or for optimization. BTW., Mathematica did the same thing. Very handy if you are doing symbolic operations on polynomials, and you can easily keep exact coefficients for a long time. Another point is that a long division would never give an overflow. Overflow since floats are too limited is an effect I don't find funny. ciao - chris --=20 Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido@python.org Thu Oct 3 01:18:28 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 02 Oct 2002 20:18:28 -0400 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Wed, 02 Oct 2002 18:14:21 EDT." <20021002221421.GA29958@thyrsus.com> References: <20021002221421.GA29958@thyrsus.com> Message-ID: <200210030018.g930IT006079@pcp02138704pcs.reston01.va.comcast.net> > python-pep@ccraig.org : > > I just uploaded a reference implementation of how rationals might look > > in Python as patch 617779 [1]. I do have some new issues for > > discussion that I'd like to get some comments on before I change the > > PEP. > > > > 1) Should future division return rationals rather than floats. I had > > sort of assumed this would happen, but I just had a discussion with > > Kirby Urner and couldn't convince him it was a good idea, so I guess > > it isn't so clear. > > > > Arguments for: > > - you don't lose precision on divides > > - It provides a really nice way to specify rationals (i.e. 1/3) > > - It allows you to eventually unify int/long/rationals so that > > rationals with a denominator of 1 are automagically upcast. > > > > Arguments against: > > - people who have already changed their code to expect floats will > > have to change it again > > - rationals are slow [ESR] > +1 for returning rationals. It's the right thing -- and if it fails, > it will fail noisily, right? Not clear at all. ABC did this, and we found that a common problem was that a program doing numeric stuff would run very slowly (i.e. the opposite of failing noisily). How are you going to print rationals? If str(1/3) (and hence print 1/3) or repr(1/3) will return "1/3", that is surely going to cause a huge amount of breakage. > > 2) Should floats compare equal with rationals only when they are > > equal, or whenever the are the closest float? (i.e. will .2 > > compare equal to rational(1, 5)) > > > > 3) Should rationals try to hash the same as floats? My leaning on > > this is that it will be decided by (2). If they compare equal when > > 'close enough' then they should hash the same, if not then they should > > only hash the same when both are integral. I would rather not see .5 > > hash with rational(1, 2) but not .2 with rational(1, 5). > > APL faced this problem twenty-five years ago. I like its solution; > a `fuzz' variable defining the close-enough-for-equality range. It's not at all clear that that's the right solution. The problem with a fuzz variable is that there's only one, and library modules may end up fighting over the right value for what they are trying to accomplish. Making it a per-module variable causes the opposite set of problems. I'm all for adding rationals to the language -- but I'd like them segregated until we have a lot more experience with how they behave. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Oct 3 01:20:11 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 02 Oct 2002 20:20:11 -0400 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Wed, 02 Oct 2002 18:27:22 EDT." References: <20021002221421.GA29958@thyrsus.com> Message-ID: <200210030020.g930KBT06090@pcp02138704pcs.reston01.va.comcast.net> [Andrew Koenig] > Much as I like APL, I'd rather use Scheme's numeric model. I've heard that before, but I've also heard criticism of Scheme's numeric model. "It works in Scheme" doesn't give me the warm fuzzy feeling that it's been tried in real life. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Oct 3 01:22:52 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 02 Oct 2002 20:22:52 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Wed, 02 Oct 2002 18:56:49 EDT." References: <20021002221421.GA29958@thyrsus.com> Message-ID: <200210030022.g930Mrl06112@pcp02138704pcs.reston01.va.comcast.net> > But those feelings are no proof of anything. How do we get the > confirmation that using rationals in Python would be easy going and > innocuous in practice, beforehand? By adding them to the language but as an isolated type. The right conversions should happen when you mix rationals with other types of numbers (int/long -> rational -> float -> complex), but no operations should return rationals unless a rational goes in. --Guido van Rossum (home page: http://www.python.org/~guido/) From greg@cosc.canterbury.ac.nz Thu Oct 3 01:42:56 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Oct 2002 12:42:56 +1200 (NZST) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030022.g930Mrl06112@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> Guido: > The right conversions should happen when you mix rationals with other > types of numbers (int/long -> rational -> float -> complex), but no > operations should return rationals unless a rational goes in. Maybe there should be a separate operator for rational division? 1/3 --> float 1//3 --> int 1///3 --> rational (Okay, a 3-char operator is a bit verbose, but I can't think of anything else that looks division-like just at the moment.) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From ark@research.att.com Thu Oct 3 02:10:52 2002 From: ark@research.att.com (Andrew Koenig) Date: Wed, 2 Oct 2002 21:10:52 -0400 (EDT) Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030020.g930KBT06090@pcp02138704pcs.reston01.va.comcast.net> (message from Guido van Rossum on Wed, 02 Oct 2002 20:20:11 -0400) References: <20021002221421.GA29958@thyrsus.com> <200210030020.g930KBT06090@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210030110.g931AqP07405@europa.research.att.com> Guido> [Andrew Koenig] >> Much as I like APL, I'd rather use Scheme's numeric model. Guido> I've heard that before, but I've also heard criticism of Guido> Scheme's numeric model. "It works in Scheme" doesn't give me Guido> the warm fuzzy feeling that it's been tried in real life. ...and "It works in APL" does? More seriously, there aren't that many languages with infinite-precision rationals, which means there aren't all that many precedents. I find the partial ordering among Python's types interesting. If we use "<" to mean "is a strict subset of", then int < long < rational (except perhaps on machines with 64-bit int, which opens a different can of worms entirely) int < float < rational float < complex Excluding complex, then, adding rational to the numeric types makes the numeric types a lattice. We could make all of the numeric types a lattice by adding a "complex rational" type: complex rational | \___ | \ rational complex / \ ____/ / \ / long float \ __/ \ / \ / int What's nice about a lattice is that for any two types T1 and T2, there is a unique minimum type T of which T1 and T2 are both subsets (not necessarily proper subsets, because T1 could be a subset of T2 or vice versa). From ark@research.att.com Thu Oct 3 02:17:30 2002 From: ark@research.att.com (Andrew Koenig) Date: 02 Oct 2002 21:17:30 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: References: <20021002221421.GA29958@thyrsus.com> Message-ID: By the way, has anyone considered the idea of making operations on rationals faster by never storing any trailing zeroes in the numerator or denominator? Instead, strip them and store the (signed) difference between the number of zeroes you've stripped from the numerator and the number you've stripped from the denominator. In other words, instead of storing (num, denom) and having the number be the (exact) value of num/demon, store (num, denom, exp) and have the number be the (exact) value of (2**exp)*num/denom. -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From tim.one@comcast.net Thu Oct 3 02:21:54 2002 From: tim.one@comcast.net (Tim Peters) Date: Wed, 02 Oct 2002 21:21:54 -0400 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030020.g930KBT06090@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Andrew Koenig] > Much as I like APL, I'd rather use Scheme's numeric model. [Guido] > I've heard that before, but I've also heard criticism of Scheme's > numeric model. "It works in Scheme" doesn't give me the warm fuzzy > feeling that it's been tried in real life. We've been thru this before too , but it doesn't even work in Scheme -- the Scheme std is too permissive in what it allows conforming implementations to get away (rationals aren't required; unbounded ints aren't required; ints *period* aren't required; while an "exact" flag is required, it has no portable mandatory semantics outside the (also undefined) range of numbers needed to index vectors; etc). Real number-crunchers have no use for it even in a full implementation, as it doesn't have a way to force precision-vs-space tradeoffs without extending the language. There's a reason the NumPy folks never bug you for Scheme features . From ark@research.att.com Thu Oct 3 02:22:51 2002 From: ark@research.att.com (Andrew Koenig) Date: 02 Oct 2002 21:22:51 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> References: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> Message-ID: Greg> 1/3 --> float Greg> 1//3 --> int Greg> 1///3 --> rational Greg> (Okay, a 3-char operator is a bit verbose, but I can't Greg> think of anything else that looks division-like just at Greg> the moment.) I can't resist: A one-l lama is a priest, A two-l llama is a beast, But I'll bet you a silk pyjama There isn't any three-l lllama. --Ogden Nash PS: When I first saw this poem, it was accompanied by a claim (in a footnote) that a three-l lllama is a substantial conflagration. -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From tim.one@comcast.net Thu Oct 3 02:31:11 2002 From: tim.one@comcast.net (Tim Peters) Date: Wed, 02 Oct 2002 21:31:11 -0400 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030018.g930IT006079@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido] > ... > Not clear at all. ABC did this, and we found that a common problem > was that a program doing numeric stuff would run very slowly (i.e. the > opposite of failing noisily). Time for my yearly repetition of that I believe that, at least for me, many (perhaps most, but not all) of the surprises in ABC were due to that "floating-point literals" (like 6.02e23) were also treated as exact rationals. > ... > The problem with a fuzz variable is that there's only one, and > library modules may end up fighting over the right value for what > they are trying to accomplish. Making it a per-module variable > causes the opposite set of problems. Knuth defines a much more sophisticated notion of "fuzzy comparison" for floats (TAoCP Vol 2). That never caught on either. I'm never clear on what people think they're *solving* with suggestions like these -- binary floating-point is so horrendously at odds with "common sense" regardless that it solves nothing. In APL there was a particular reason for it, in order to create arrays of booleans from whole-array comparison operators efficiently, but that didn't make it a useful *scalar* gimmick there either, and in my brief APL days the damn fuzz destroyed more careful numeric algorithms than it helped for that reason. > I'm all for adding rationals to the language -- but I'd like them > segregated until we have a lot more experience with how they behave. They behave most like precocious children with voracious appetite . From tim.one@comcast.net Thu Oct 3 02:44:27 2002 From: tim.one@comcast.net (Tim Peters) Date: Wed, 02 Oct 2002 21:44:27 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Message-ID: [Andrew Koenig] > By the way, has anyone considered the idea of making operations > on rationals faster by never storing any trailing zeroes in the > numerator or denominator? Instead, strip them and store the > (signed) difference between the number of zeroes you've stripped > from the numerator and the number you've stripped from the denominator. > > In other words, instead of storing (num, denom) and having the number > be the (exact) value of num/demon, store (num, denom, exp) and have the > number be the (exact) value of (2**exp)*num/denom. How many trailing zeroes do you expect to save this way? "On average", a random int has exactly n (>= 0) trailing zero bits with probability 1/2**(n+1). Then how expensive is it for operations to keep track of them appropriately (esp. + and -, where the split representation seems offhand (to me) unnatural to work with)? My FixedPoint.py does a variation of this for a form of unnormalized rational, except the "trailing zeroes" there are wrt base 10. But it wasn't to save time (heh) in that context, it was to create a faithful illusion of exact decimal arithmetic. From ark@research.att.com Thu Oct 3 03:16:07 2002 From: ark@research.att.com (Andrew Koenig) Date: Wed, 2 Oct 2002 22:16:07 -0400 (EDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: (message from Tim Peters on Wed, 02 Oct 2002 21:44:27 -0400) References: Message-ID: <200210030216.g932G7707727@europa.research.att.com> Tim> How many trailing zeroes do you expect to save this way? The real point is to have small, efficient rational representations for floating-point numbers, even those with large exponents. From greg@cosc.canterbury.ac.nz Thu Oct 3 03:18:59 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Oct 2002 14:18:59 +1200 (NZST) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Message-ID: <200210030218.g932IxH20780@oma.cosc.canterbury.ac.nz> Andrew Koenig : > A one-l lama is a priest, > A two-l llama is a beast, > But I'll bet you a silk pyjama > There isn't any three-l lllama. > > --Ogden Nash > > PS: When I first saw this poem, it was accompanied by a claim (in a > footnote) that a three-l lllama is a substantial conflagration. :-) !!! I have a vision of an obscure corner of Tibet featuring an order of special firefighting lamas, ready at a moment's notice to jump on their llamas and race off to do battle with a lllama... And whenever I see "1/3" now I'm going to want to pronouce it "one lama three". What have you done to me? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg@cosc.canterbury.ac.nz Thu Oct 3 03:22:30 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Oct 2002 14:22:30 +1200 (NZST) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030216.g932G7707727@europa.research.att.com> Message-ID: <200210030222.g932MUd20787@oma.cosc.canterbury.ac.nz> Andrew Koenig : > The real point is to have small, efficient rational representations > for floating-point numbers, even those with large exponents. But how long are they going to *stay* small and efficient, once you start doing arithmetic on them? The problem as I see it is not the magnitude of the number, but that the number of significant bits grows without bound, if you never throw any of them away. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From ark@research.att.com Thu Oct 3 03:25:07 2002 From: ark@research.att.com (Andrew Koenig) Date: 02 Oct 2002 22:25:07 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030222.g932MUd20787@oma.cosc.canterbury.ac.nz> References: <200210030222.g932MUd20787@oma.cosc.canterbury.ac.nz> Message-ID: Greg> But how long are they going to *stay* small and efficient, Greg> once you start doing arithmetic on them? Depends on what kind of arithmetic. Greg> The problem as I see it is not the magnitude of the number, but Greg> that the number of significant bits grows without bound, if you Greg> never throw any of them away. So I'm suggesting that maybe it won't get bad as quickly if you keep only the significant bits :-) Seriously, I don't know whether it would help in practice. It might be that normalizing rationals from time to time would be enough. -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From esr@thyrsus.com Thu Oct 3 03:41:21 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Wed, 2 Oct 2002 22:41:21 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030218.g932IxH20780@oma.cosc.canterbury.ac.nz> References: <200210030218.g932IxH20780@oma.cosc.canterbury.ac.nz> Message-ID: <20021003024121.GA31589@thyrsus.com> Greg Ewing : > Andrew Koenig : > > > A one-l lama is a priest, > > A two-l llama is a beast, > > But I'll bet you a silk pyjama > > There isn't any three-l lllama. > > > > --Ogden Nash > > > > PS: When I first saw this poem, it was accompanied by a claim (in a > > footnote) that a three-l lllama is a substantial conflagration. > > :-) !!! > > I have a vision of an obscure corner of Tibet featuring > an order of special firefighting lamas, ready at a moment's > notice to jump on their llamas and race off to do battle > with a lllama... :-) By the way, the footnote read as follows: "The author's attention has been called to a type of conflagration known as a three-alarmer. Pooh." > And whenever I see "1/3" now I'm going to want to > pronouce it "one lama three". What have you done to > me? Oh, ghods. You realize this is turning into a classic Jargon File entry, don't you? @hd{lama} @g{n.} [Python] The division operator /; thus 1/3 is pronounced "one lama three". Coined during and October 2002 on the Python-development mailing list. Someone proposed that /, //, and /// should stand for integer-, rational-, and float-valued division, and someone else quoted Ogden Nash's poem @uref{http://www.cs.rice.edu/~ssiyer/minstrels/poems/1080.html, The Lama}. Amidst talk of firefighting llamas in Tibet, the analogy stuck. Of course, I can't actually add this. The last three words aren't true. Yet.,, -- Eric S. Raymond From python@rcn.com Thu Oct 3 03:50:56 2002 From: python@rcn.com (Raymond Hettinger) Date: Wed, 2 Oct 2002 22:50:56 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <20021002221421.GA29958@thyrsus.com> Message-ID: <009401c26a87$b2e0c780$b2e97ad1@othello> [François Pinard] > While I agree with the theoretical arguments, I have the practical fear that > rationals could grow very big, rather quickly, in the course of a long > computation involving them in various ways. By big, I mean the numerator > and denominator of the fraction taken in isolation, not the number itself. > Consider inversions of an integer matrices, approximations with truncated > series, or worse things like, maybe, discrete Fourier transforms. > > Bigger rationals are, slower they become, and more memory they take. The > danger is that programmers may get surprised or hurt by Python performance > degradation, raising frequent and recurrent questions here and elsewhere. There should be a builtin variable (overriddable within some inner scope) for a maximum denominator magnitude. It should default to some value where performance tanks. If set to None, then no limit would apply. The HP32SII calculator implements a useful rational model using flags and a maximum denominator register. If the first flag is clear, fractions are have denominators upto the maximum value. If only the first flag is set, fractions always use the maximum denominator as the denominator and are then reduced (i.e. if the max is 8, then .5 is represented as 1/2 and .1 is represented as 1/8). [Christopher A. Craig] > > 3) Should rationals try to hash the same as floats? My leaning on > > this is that it will be decided by (2). If they compare equal when > > 'close enough' then they should hash the same, if not then they should > > only hash the same when both are integral. I would rather not see .5 > > hash with rational(1, 2) but not .2 with rational(1, 5). [Eric Raymond] > APL faced this problem twenty-five years ago. I like its solution; > a `fuzz' variable defining the close-enough-for-equality range. Instead of a global fuzz variable, I would prefer a fuzzy compare function with a specifiable fuzz factor and a reasonable default setting. This approach is more explicit in leaving == as an exact compare, specifying nearlyequal(a,b) when that is what is meant, and providing a locally specifiable factor on each compare, for example, nearlyequal(a,b, absolute=1e-7). Also, a fuzzy compare function could be set to use absolute or relative differences when needed. For example: if nearlyequal(a,b, relative=.01): # are a and b within 1% Raymond Hettinger From tismer@tismer.com Thu Oct 3 03:55:25 2002 From: tismer@tismer.com (Christian Tismer) Date: Wed, 02 Oct 2002 19:55:25 -0700 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <200210030222.g932MUd20787@oma.cosc.canterbury.ac.nz> Message-ID: <3D9BB19D.4000401@tismer.com> Andrew Koenig wrote: > Greg> But how long are they going to *stay* small and efficient, > Greg> once you start doing arithmetic on them? > > Depends on what kind of arithmetic. I doubt very much that this would help in more than few simple cases. Floats with high exponents are usually incorrect, since the number of bits of the exact representation is already truncated to the float limitation. With an exact rational representation all the bits are preserved, and it is unlikely that many of the lower bits simply become zero. > Greg> The problem as I see it is not the magnitude of the number, but > Greg> that the number of significant bits grows without bound, if you > Greg> never throw any of them away. > > So I'm suggesting that maybe it won't get bad as quickly if > you keep only the significant bits :-) Yes. And this is what a rounding to float would do for you. > Seriously, I don't know whether it would help in practice. > It might be that normalizing rationals from time to time would > be enough. Ok, not just zeros, but normalizing the fraction to no common denominator. That makes sense if lots of small numbers/fractions were multiplied and prime factors pile up. Not so with addition. This creates completely different prime factors all the time, and addition becomes much more expensive than multiplication when the common denominator must always be reduced to the minimum. gotta-be-expensive -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From python@rcn.com Thu Oct 3 04:02:58 2002 From: python@rcn.com (Raymond Hettinger) Date: Wed, 2 Oct 2002 23:02:58 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <200210030222.g932MUd20787@oma.cosc.canterbury.ac.nz> <3D9BB19D.4000401@tismer.com> Message-ID: <00ac01c26a89$6117aac0$b2e97ad1@othello> From: "Christian Tismer" > > Greg> The problem as I see it is not the magnitude of the number, but > > Greg> that the number of significant bits grows without bound, if you > > Greg> never throw any of them away. > > > > So I'm suggesting that maybe it won't get bad as quickly if > > you keep only the significant bits :-) > > Yes. And this is what a rounding to float would do for you. That is unless you wanted more precision than comes with a float. Normalization to some arbitrary maximum denominator magnitude would be a more general purpose solution. Raymond Hettinger From greg@cosc.canterbury.ac.nz Thu Oct 3 04:07:19 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Oct 2002 15:07:19 +1200 (NZST) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <009401c26a87$b2e0c780$b2e97ad1@othello> Message-ID: <200210030307.g9337JW21043@oma.cosc.canterbury.ac.nz> Raymond Hettinger : > The HP32SII calculator implements a useful rational model using > flags and a maximum denominator register. If the first flag is clear, > fractions are have denominators upto the maximum value. If only > the first flag is set, fractions always use the maximum denominator > as the denominator and are then reduced (i.e. if the max is 8, then > .5 is represented as 1/2 and .1 is represented as 1/8). I can see that being useful when you're doing calculations with, e.g. measurements in 64ths of an inch and you don't care if anything smaller than that isn't quite exact. But I get the impression that the folks who want rationals in Python want them precisely because they're *always* exact. If you couldn't rely on them to always be exact, it would defeat the purpose. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From fschaef@ces.clemson.edu Thu Oct 3 04:12:35 2002 From: fschaef@ces.clemson.edu (Frank S.) Date: Wed, 2 Oct 2002 23:12:35 -0400 (EDT) Subject: [Python-Dev] RFC: Option Parsing Libraries Message-ID: After releasing the GetPot 1.0 Version I did some research on other packages and I found the the Option Parsing RFC. I totally missed the discussion. How come that nobody contacted me about the GetPot package ? To get a python doc out should not be a problem at all. Not one single person, though, complained until now, since the C++ doc is totally equivalent. Take a look at http://getpot.sourceforge.net for new features of the 1.0 Version. Except for the automated documenting feature (which may be a little intrusive in its format) I don't really see what Optik brings new on the table. Guys, even though GetPot is designed for many languages, I'd like to join the command line parsing discussion. Best Regards, Frank Schaefer. From greg@cosc.canterbury.ac.nz Thu Oct 3 04:14:24 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Oct 2002 15:14:24 +1200 (NZST) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021003024121.GA31589@thyrsus.com> Message-ID: <200210030314.g933EOb21057@oma.cosc.canterbury.ac.nz> "Eric S. Raymond" : > Someone proposed that /, //, and /// should stand for > integer-, rational-, and float-valued division, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ actually, float-, integer- and rational- I'm not suggesting the existing ones should be changed! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tismer@tismer.com Thu Oct 3 04:14:34 2002 From: tismer@tismer.com (Christian Tismer) Date: Wed, 02 Oct 2002 20:14:34 -0700 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <20021002221421.GA29958@thyrsus.com> <009401c26a87$b2e0c780$b2e97ad1@othello> Message-ID: <3D9BB61A.5070902@tismer.com> Raymond Hettinger wrote: ... > There should be a builtin variable (overriddable within some inner scope) > for a maximum denominator magnitude. It should default to some value > where performance tanks. If set to None, then no limit would apply. > > The HP32SII calculator implements a useful rational model using > flags and a maximum denominator register. If the first flag is clear, > fractions are have denominators upto the maximum value. If only > the first flag is set, fractions always use the maximum denominator > as the denominator and are then reduced (i.e. if the max is 8, then > .5 is represented as 1/2 and .1 is represented as 1/8). Makes very much sense to me. -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From ark@research.att.com Thu Oct 3 04:14:12 2002 From: ark@research.att.com (Andrew Koenig) Date: Wed, 2 Oct 2002 23:14:12 -0400 (EDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <3D9BB19D.4000401@tismer.com> (message from Christian Tismer on Wed, 02 Oct 2002 19:55:25 -0700) References: <200210030222.g932MUd20787@oma.cosc.canterbury.ac.nz> <3D9BB19D.4000401@tismer.com> Message-ID: <200210030314.g933ECr08046@europa.research.att.com> >> So I'm suggesting that maybe it won't get bad as quickly if >> you keep only the significant bits :-) Christian> Yes. And this is what a rounding to float would do for you. Except that doing so would throw away some significant bits too. From python@rcn.com Thu Oct 3 04:13:16 2002 From: python@rcn.com (Raymond Hettinger) Date: Wed, 2 Oct 2002 23:13:16 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <200210030307.g9337JW21043@oma.cosc.canterbury.ac.nz> Message-ID: <00fb01c26a8b$91f5d200$b2e97ad1@othello> From: "Greg Ewing" > > The HP32SII calculator implements a useful rational model using > > flags and a maximum denominator register. If the first flag is clear, > > fractions are have denominators upto the maximum value. If only > > the first flag is set, fractions always use the maximum denominator > > as the denominator and are then reduced (i.e. if the max is 8, then > > .5 is represented as 1/2 and .1 is represented as 1/8). > > I can see that being useful when you're doing calculations > with, e.g. measurements in 64ths of an inch and you don't > care if anything smaller than that isn't quite exact. > > But I get the impression that the folks who want rationals > in Python want them precisely because they're *always* > exact. If you couldn't rely on them to always be exact, > it would defeat the purpose. For that, I propose maxdenom=None to let the rationals grow without bound. Raymond Hettinger From greg@cosc.canterbury.ac.nz Thu Oct 3 04:22:50 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Oct 2002 15:22:50 +1200 (NZST) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021003024121.GA31589@thyrsus.com> Message-ID: <200210030322.g933MoR21076@oma.cosc.canterbury.ac.nz> "Eric S. Raymond" : > By the way, the footnote read as follows: > > "The author's attention has been called to a type of conflagration known > as a three-alarmer. Pooh." This is getting weird. I did a Google search for "lllama" and found this, apparently related to some sort of roleplaying game: From: Alex Yeager (YeagerAW@maritz.com) Date: Mon, 25 May 1998 Type: Plot Name: Three-L Lllama Graphic: A building on fire. Quote: Ogden Nash was wrong! They DO exist! Text: Disaster! This is an Instant Attack to Destroy any Place. Its Power is 12, increased by four for every undrawn Plot you throw onto the blaze. If the Place is Coastal, it may spend its token to defend itself. If the attack succeeds, the target is Devastated. If it succeeds by more than 6, the target is destroyed! Quote2: So where do I claim those silk pajamas?... Requirements: none Illamanated Conspiracy Proposed by "Yeager, Alex" (YeagerAW@Maritz.com) on Mon, 25 May 1998. So, was this Lllama spell inspired by the "three-alarmer" pun, or is it just a strange coincidence? Or is it some secret plot by the PSU to stop us sleeping at nights? Inquiring minds probably don't really want to know... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg@cosc.canterbury.ac.nz Thu Oct 3 04:24:16 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Oct 2002 15:24:16 +1200 (NZST) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <00ac01c26a89$6117aac0$b2e97ad1@othello> Message-ID: <200210030324.g933OGx21083@oma.cosc.canterbury.ac.nz> Raymond Hettinger : > That is unless you wanted more precision than comes with a float. > Normalization to some arbitrary maximum denominator magnitude > would be a more general purpose solution. Wouldn't a float type with specifiable precision do just as well for that, and be more efficient? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg@cosc.canterbury.ac.nz Thu Oct 3 04:27:51 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Oct 2002 15:27:51 +1200 (NZST) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <00fb01c26a8b$91f5d200$b2e97ad1@othello> Message-ID: <200210030327.g933Rpw21328@oma.cosc.canterbury.ac.nz> Raymond Hettinger : > For that, I propose maxdenom=None to let the rationals > grow without bound. A global setting for this would have all the same problems we've just gone over in relation to the "fuzz" setting. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one@comcast.net Thu Oct 3 05:00:51 2002 From: tim.one@comcast.net (Tim Peters) Date: Thu, 03 Oct 2002 00:00:51 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Message-ID: [Andrew Koenig] > ... > Seriously, I don't know whether it would help in practice. > It might be that normalizing rationals from time to time would > be enough. Do check out Val=E9rie M=E9nissier-Morain's work on exact rational ar= ithmetic in Caml; I've referenced it several times before, and this discussion is= always the same . In real apps, the time it takes to normalize ration= als can be a disaster (the gcd of two random ints is most likely to be 1, and longint gcds are very expensive). In other real apps, not normalizin= g rationals can be a disaster. There isn't a one-size-fits-all policy = for this, and a serious implementation has to take that seriously (sorry,= Scheme ). From Anthony Baxter Thu Oct 3 05:01:28 2002 From: Anthony Baxter (Anthony Baxter) Date: Thu, 03 Oct 2002 14:01:28 +1000 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030322.g933MoR21076@oma.cosc.canterbury.ac.nz> Message-ID: <200210030401.g9341Sb16764@localhost.localdomain> >>> Greg Ewing wrote > This is getting weird. I did a Google search for "lllama" > and found this, apparently related to some sort of roleplaying > game: > > From: Alex Yeager (YeagerAW@maritz.com) > Date: Mon, 25 May 1998 > > Type: Plot > Name: Three-L Lllama > Graphic: A building on fire. > Quote: Ogden Nash was wrong! They DO exist! This is a card for Steve Jackson's Illuminati: New World Order collectible card game. A lot of fun, but don't play it with people who take badly to being brutally betrayed and backstabbed (that's kinda the point of the game). Anthony -- Anthony Baxter It's never too late to have a happy childhood. From oren-py-d@hishome.net Thu Oct 3 06:28:29 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Thu, 3 Oct 2002 01:28:29 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030216.g932G7707727@europa.research.att.com> References: <200210030216.g932G7707727@europa.research.att.com> Message-ID: <20021003052829.GA59708@hishome.net> On Wed, Oct 02, 2002 at 10:16:07PM -0400, Andrew Koenig wrote: > Tim> How many trailing zeroes do you expect to save this way? > > The real point is to have small, efficient rational representations > for floating-point numbers, even those with large exponents. But why would you want to convert floating point numbers to rationals in the first place? Floating point numbers are generally inexact. Converting them to exact rationals can give a false sense of exactitude. Any calculation involving a floating point number should taint the result with inexactitude and yield a floating point result. Oren From aleax@aleax.it Thu Oct 3 06:56:50 2002 From: aleax@aleax.it (Alex Martelli) Date: Thu, 3 Oct 2002 07:56:50 +0200 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> References: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> Message-ID: On Thursday 03 October 2002 02:42 am, Greg Ewing wrote: ... > 1///3 --> rational > > (Okay, a 3-char operator is a bit verbose, but I can't > think of anything else that looks division-like just at > the moment.) 1\3 ... ? Alex From tim.one@comcast.net Thu Oct 3 07:01:49 2002 From: tim.one@comcast.net (Tim Peters) Date: Thu, 03 Oct 2002 02:01:49 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021003052829.GA59708@hishome.net> Message-ID: [Oren Tirosh] > But why would you want to convert floating point numbers to rationals > in the first place? For one thing, to continue an exact calculation that has grown too large to continue in floating-point without losing information, but where the extreme expense of using rationals instead of floating point pushes you toward using fp just as long as you can. > Floating point numbers are generally inexact. The IEEE-754 standard mandates an "inexact flag" (and your HW has one) for this very kind of purpose. Your FPU knows for certain whether you've ever lost a bit of information when doing basic fp operations; unfortunately, it's still extremely clumsy to get at this info. > Converting them to exact rationals can give a false sense of > exactitude. So can printing more digits than warranted . > Any calculation involving a floating point number should taint > the result with inexactitude and yield a floating point result. How old are you? See -- integers can be inexact too. Representation has nothing to do with exactness, which is one part of this story Scheme got straight. From Anthony Baxter Thu Oct 3 07:03:20 2002 From: Anthony Baxter (Anthony Baxter) Date: Thu, 03 Oct 2002 16:03:20 +1000 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Message-ID: <200210030603.g9363Kd17878@localhost.localdomain> >>> Alex Martelli wrote > On Thursday 03 October 2002 02:42 am, Greg Ewing wrote: > ... > > 1///3 --> rational > > > > (Okay, a 3-char operator is a bit verbose, but I can't > > think of anything else that looks division-like just at > > the moment.) > > 1\3 ... ? a = """ 1 - 3""" From oren-py-d@hishome.net Thu Oct 3 07:10:45 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Thu, 3 Oct 2002 02:10:45 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: References: Message-ID: <20021003061045.GB59708@hishome.net> On Thu, Oct 03, 2002 at 12:00:51AM -0400, Tim Peters wrote: > [Andrew Koenig] > > ... > > Seriously, I don't know whether it would help in practice. > > It might be that normalizing rationals from time to time would > > be enough. > > Do check out Valérie Ménissier-Morain's work on exact rational arithmetic in > Caml; I've referenced it several times before, and this discussion is always > the same . In real apps, the time it takes to normalize rationals can > be a disaster (the gcd of two random ints is most likely to be 1, and > longint gcds are very expensive). In other real apps, not normalizing > rationals can be a disaster. There isn't a one-size-fits-all policy for > this, and a serious implementation has to take that seriously (sorry, Scheme > ). Here is a quick attempt at a one-size-fits-all policy: Rationals have a denormalized internal representation but always appear to be normalized. Rationals may be normalized in-place and still be considered immutable because this is invisible to the user. Rationals are normalized in-place in the following cases: 1. When the string representation of the rational number is required 2. When accessing the numerator or denominator explicitly 3. Occasionally, according to some magic heuristics to prevent growth The price for trying to please everybody is complexity but performance does not have to suffer too much. Oren From oren-py-d@hishome.net Thu Oct 3 08:21:47 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Thu, 3 Oct 2002 03:21:47 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: References: <20021003052829.GA59708@hishome.net> Message-ID: <20021003072147.GC59708@hishome.net> On Thu, Oct 03, 2002 at 02:01:49AM -0400, Tim Peters wrote: > [Oren Tirosh] > > But why would you want to convert floating point numbers to rationals > > in the first place? > > For one thing, to continue an exact calculation that has grown too large to > continue in floating-point without losing information, but where the extreme > expense of using rationals instead of floating point pushes you toward using > fp just as long as you can. > > > Floating point numbers are generally inexact. > > The IEEE-754 standard mandates an "inexact flag" (and your HW has one) for > this very kind of purpose. Your FPU knows for certain whether you've ever > lost a bit of information when doing basic fp operations; unfortunately, > it's still extremely clumsy to get at this info. The IEEE-754 standard also mandates NaNs, Infs, rounding modes and several other features. These are not supported by Python in a consistent and portable manner nor it is likely that they can be supported on all Python platforms in the foreseeable future because of the limitations of the underlying C libraries. > > Converting them to exact rationals can give a false sense of > > exactitude. > > So can printing more digits than warranted . Yes, my material strength teacher drove this point home by taking off points for results with either too many or too few digits. > > Any calculation involving a floating point number should taint > > the result with inexactitude and yield a floating point result. > > How old are you? See -- integers can be inexact too. Representation has > nothing to do with exactness, which is one part of this story Scheme got > straight. I am approaximate 61131/76 fortnights old :-) A integer is always exact and integer arithmetic is always exact with the exception of division. The mapping of this mathematical integer to some real-world meaning may be inexact, though. I won't sue anyone because the pack of rubber bands doesn't contain exactly 1000 pieces. A double precision floating point number may represent some number exactly and produce exact results for some arithmetic operations but in practice if I type 0.3 and get 0.29999999999999999 the numbers aren't exact in the first place and if I have no access to the flags that tell me if the result is exact or not I must treat all results as inexact. As usual, your arguments are flawless - in an academic sort of way. In practice, though, I think my original point about inexactness of of floats and tainting of results still stands. Oren From mwh@python.net Thu Oct 3 10:14:09 2002 From: mwh@python.net (Michael Hudson) Date: 03 Oct 2002 10:14:09 +0100 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Guido van Rossum's message of "Wed, 02 Oct 2002 20:20:11 -0400" References: <20021002221421.GA29958@thyrsus.com> <200210030020.g930KBT06090@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <2mlm5fuboe.fsf@starship.python.net> Guido van Rossum writes: > [Andrew Koenig] > > Much as I like APL, I'd rather use Scheme's numeric model. > > I've heard that before, but I've also heard criticism of Scheme's > numeric model. "It works in Scheme" doesn't give me the warm fuzzy > feeling that it's been tried in real life. Could try aping Common Lisp. It's much more tightly specified, and actually has had some exposure to the real world. Here would seem to be a start: http://www.lispworks.com/reference/HyperSpec/Body/12_ac.htm Cheers, M. -- how am I expected to quit smoking if I have to deal with NT every day -- Ben Raia From esteban@ccpgames.com Thu Oct 3 11:09:04 2002 From: esteban@ccpgames.com (Esteban U.C.. Castro) Date: Thu, 3 Oct 2002 10:09:04 -0000 Subject: [Python-Dev] re: *very* revised proposal for interfaces Message-ID: <25A39AFEB31B06408C675CEF28199E5B073CAC@postur.ccp.cc> > For my purposes, I'm not sure the error checking is really a Good Thing > after some of the things Esteban Castro pointed out. If you fail to > implement the nesessary methods, that's your own problem. This approach > is a lot less work for me, and handles strange cases of late binding > more gracefully. Of course a warning would be helpful in the 95% of > cases where the assumptions made for error checking are correct. I think you misinterpret me here. I am not against checking an interface.=20 Indeed, checking would be the only way to detect implicit interface=20 compliance (one of my main points). >>> from interfaces import ListInterface >>> a =3D [1,2,3] >>> implements(a, ListInterface) 1 >>> implements(a[0], ListInterface) 0 >>> import my >>> b =3D my.SuperDuperFakeList() >>> implements(b, ListInterface) 1 It should be possible to define ListInterface once and check it against any arbitrary object, without having to fiddle with, or provide adapters for the type/class of each object that does (de facto) implement it.=20 For example, I don't want to have to do __bindings__ on the list type,=20 UserList, any extension types designed to be used as lists, wrappers or=20 proxies on lists that are not instances of UserList, etc.=20 Declaring that a class implements an interface (this is, it promises that=20 all its instances will satisfy it) still makes sense: * For documentation: declaring compliance with an interface gives a very useful hint on the purpose and usage of the class. * For safety: you may be afraid that satisfying some method signatures alone=20 doesn't guarantee that an object is safe to use in contexts where your interface is required.=20 For example, you have this my.Shape interface with this draw() method. There=20 are too many classes around that provide a draw() method and which won't=20 work for the purposes of my.Shape (and that's why a simple check for draw() is not enough either). Implicit interface satisfaction is not a concern here,=20 because it doesn't make sense that anyone has implemented useful my.Shapes=20 before knowing about your module. So it makes full sense for you to require=20 that any objects that want to implement my.Shape explicitly declare so. This=20 does not necessarily replace checking for draw(). The author of each specific=20 interface must decide her policy on this. * For performance: you may want to cache the results of class checks. You may even do without checking at all, but I don't think this is a good idea. I have my own ideas on how the syntax should be but I'd rather bounce them and=20 let them mature off this list. If anyone is willing to put up a mailing list or=20 just discuss these things privately, please let me know. Esteban. From skip@pobox.com Thu Oct 3 13:30:12 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 3 Oct 2002 07:30:12 -0500 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <3D9BB19D.4000401@tismer.com> References: <200210030222.g932MUd20787@oma.cosc.canterbury.ac.nz> <3D9BB19D.4000401@tismer.com> Message-ID: <15772.14420.986230.807066@localhost.localdomain> Danger Will Robinson! I know next to nothing about numerical analysis. I don't even play a numerical analyst on TV. >> So I'm suggesting that maybe it won't get bad as quickly if you keep >> only the significant bits :-) Christian> Yes. And this is what a rounding to float would do for you. Okay, we have a tension here. On the one hand, float representation provides the convenience of bounded storage at the expense of loss of precision. Rationals, on the other hand, have the potential to prevent loss of precision, but at the cost of potentially unbounded storage (and greatly increased computational cost). Is there a middle ground where you can say, in effect, "give me rational numbers which are truncated to provide precision no worse than N-bit floating point numbers"? I'm sure that's stated poorly (and incorrectly), so here's an example. Pi is an irrational number, represented in Python (at least on the platform closest to hand) as 3.1415926535897931. Suppose I needed pi to this precision though: 3.14159265358979323846264. Could I somehow parameterize rational numbers to give me the required precision - that is, numerators and/or denominators were automatically truncated to provide the desired precision and prevent unbounded growth (space and computational)? Skip From sholden@holdenweb.com Thu Oct 3 13:52:01 2002 From: sholden@holdenweb.com (Steve Holden) Date: Thu, 3 Oct 2002 08:52:01 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <200210030222.g932MUd20787@oma.cosc.canterbury.ac.nz> <3D9BB19D.4000401@tismer.com> Message-ID: <00bf01c26adb$abf530b0$8701a8c0@holdenweb.com> Christian Tismer wrote ... > Andrew Koenig wrote: [rational representations] > > Seriously, I don't know whether it would help in practice. > > It might be that normalizing rationals from time to time would > > be enough. > > Ok, not just zeros, but normalizing the fraction to no > common denominator. That makes sense if lots of small > numbers/fractions were multiplied and prime factors > pile up. > Not so with addition. This creates completely different > prime factors all the time, and addition becomes much > more expensive than multiplication when the common > denominator must always be reduced to the minimum. > > gotta-be-expensive -- chris > Next someone will suggest that we store rationals as a sequence of coefficients of the prime factors. This would make primes really easy to recognise (since they'd be a sequence of zeros followed by a one with a single one as denominator). but-it-might-slow-down-other-operations-a-bit-ly y'rs - steve ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Previous .sig file retired to www.homeforoldsigs.com ----------------------------------------------------------------------- From skip@pobox.com Thu Oct 3 14:02:23 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 3 Oct 2002 08:02:23 -0500 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030327.g933Rpw21328@oma.cosc.canterbury.ac.nz> References: <00fb01c26a8b$91f5d200$b2e97ad1@othello> <200210030327.g933Rpw21328@oma.cosc.canterbury.ac.nz> Message-ID: <15772.16351.770660.727862@localhost.localdomain> Greg> Raymond Hettinger : >> For that, I propose maxdenom=None to let the rationals grow without >> bound. Greg> A global setting for this would have all the same problems we've Greg> just gone over in relation to the "fuzz" setting. If it was a builtin, different modules could simply set their own value at global or local levels to override the default for the arithmetic they do. Would that make numerical stability problems nearly impossible to debug? Skip From ark@research.att.com Thu Oct 3 14:05:03 2002 From: ark@research.att.com (Andrew Koenig) Date: Thu, 3 Oct 2002 09:05:03 -0400 (EDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021003052829.GA59708@hishome.net> (message from Oren Tirosh on Thu, 3 Oct 2002 01:28:29 -0400) References: <200210030216.g932G7707727@europa.research.att.com> <20021003052829.GA59708@hishome.net> Message-ID: <200210031305.g93D53o12559@europa.research.att.com> Oren> Any calculation involving a floating point number should taint Oren> the result with inexactitude and yield a floating point result. Even comparisons? From list-python@ccraig.org Thu Oct 3 14:09:42 2002 From: list-python@ccraig.org (Christopher A. Craig) Date: 03 Oct 2002 09:09:42 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <00bf01c26adb$abf530b0$8701a8c0@holdenweb.com> References: <200210030222.g932MUd20787@oma.cosc.canterbury.ac.nz> <3D9BB19D.4000401@tismer.com> <00bf01c26adb$abf530b0$8701a8c0@holdenweb.com> Message-ID: "Steve Holden" writes: > Next someone will suggest that we store rationals as a sequence of > coefficients of the prime factors. This would make primes really easy to > recognise (since they'd be a sequence of zeros followed by a one with a > single one as denominator). It would make the gcd really fast too, even for extremely large numbers. :-) -- Christopher A. Craig "It's the one thing I understand, I guess." Bill Gates, about BASIC From David Abrahams" <200210030216.g932G7707727@europa.research.att.com> <20021003052829.GA59708@hishome.net> <200210031305.g93D53o12559@europa.research.att.com> Message-ID: <08bb01c26ad9$6f44e9f0$6701a8c0@boostconsulting.com> From: "Andrew Koenig" > Oren> Any calculation involving a floating point number should taint > Oren> the result with inexactitude and yield a floating point result. > > Even comparisons? Well, comparisons are tainted already in some sense, because you don't know how much error there is in any given number. FWIW, Boost just accepted an interval arithmetic library which does computation with a closed range of floating point numbers, making it possible to know just how fuzzy any given computation is. ----------------------------------------------------------- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From python-pep@ccraig.org Thu Oct 3 14:20:27 2002 From: python-pep@ccraig.org (Christopher A. Craig) Date: 03 Oct 2002 09:20:27 -0400 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021002221421.GA29958@thyrsus.com> References: <20021002221421.GA29958@thyrsus.com> Message-ID: "Eric S. Raymond" writes: > APL faced this problem twenty-five years ago. I like its solution; > a `fuzz' variable defining the close-enough-for-equality range. For what it's worth, the current implementation is nothing this complicated. Like every other numeric operation, compares of rationals with floats coerce to floats. So as a result rational(x)==float(x) iff float(rational(x))==float(x). That is, of course, much more permissive than if we coerced the other way and had rational(x)==float(x) iff rational(x)==rational(float(x)). -- Christopher A. Craig "You could shoot Microsoft Office off the planet and this country would run better. You would see everyone standing around saying, 'I've got so much time now.' " Scott McNealy (CEO of Sun) From paul@prescod.net Thu Oct 3 14:49:20 2002 From: paul@prescod.net (Paul Prescod) Date: Thu, 03 Oct 2002 06:49:20 -0700 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> Message-ID: <3D9C4AE0.2060406@prescod.net> Alex Martelli wrote: > On Thursday 03 October 2002 02:42 am, Greg Ewing wrote: > ... > >> 1///3 --> rational >> >>(Okay, a 3-char operator is a bit verbose, but I can't >>think of anything else that looks division-like just at >>the moment.) > > > 1\3 ... ? I predict this will be the Perl 6 solution if they run into this problem: http://www.myfonts.com/Character00F7Style1150.html Today, it has a certain impractical elegance, which is not usually a phrase I associate with Perl. But a few years from now it may be practical. Paul Prescod From ark@research.att.com Thu Oct 3 15:40:38 2002 From: ark@research.att.com (Andrew Koenig) Date: 03 Oct 2002 10:40:38 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <08bb01c26ad9$6f44e9f0$6701a8c0@boostconsulting.com> References: <200210030216.g932G7707727@europa.research.att.com> <20021003052829.GA59708@hishome.net> <200210031305.g93D53o12559@europa.research.att.com> <08bb01c26ad9$6f44e9f0$6701a8c0@boostconsulting.com> Message-ID: >> Even comparisons? David> Well, comparisons are tainted already in some sense, because you don't know David> how much error there is in any given number. Nevertheless, there can be serious problems when comparisons are not consistent. -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From ark@research.att.com Thu Oct 3 15:45:27 2002 From: ark@research.att.com (Andrew Koenig) Date: 03 Oct 2002 10:45:27 -0400 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: References: <20021002221421.GA29958@thyrsus.com> Message-ID: Christopher> For what it's worth, the current implementation is Christopher> nothing this complicated. Like every other numeric Christopher> operation, compares of rationals with floats coerce to Christopher> floats. So as a result rational(x)==float(x) iff Christopher> float(rational(x))==float(x). Attractive as this strategy may be from an implementation viewpoint, it means that comparisons do not have the order-relation properties that algorithms such as sort expect. A similar problem exists today for comparisons between float and long, by the way. -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From guido@python.org Thu Oct 3 15:56:48 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 03 Oct 2002 10:56:48 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Thu, 03 Oct 2002 12:42:56 +1200." <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> References: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> Message-ID: <200210031456.g93Eumx20160@odiug.zope.com> After reading most of the discussion, I don't know what to do about this. There are a number of different ideas that should be reviewed separately: "classic" rationals, fixed-point decimal, a new kind of floats with settable precision... I tend to think that rationals and superfloats are sufficiently esoteric that they should probably be relegated to an extension or library module. Fixed-point decimals have some nice properties, but apparently implementing it isn't easy -- Aahz has been sitting on a prototype approximately forever. I hear that Tim's FixedPoint package is now a SF project. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Oct 3 16:15:35 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 03 Oct 2002 11:15:35 -0400 Subject: [Python-Dev] RFC: Option Parsing Libraries In-Reply-To: Your message of "Wed, 02 Oct 2002 23:12:35 EDT." References: Message-ID: <200210031515.g93FFZU20358@odiug.zope.com> > After releasing the GetPot 1.0 Version I did some research on other > packages and I found the the Option Parsing RFC. What's the Option Parsing RFC? > I totally missed the discussion. How come that nobody contacted me > about the GetPot package ? Because we didn't know you existed? > To get a python doc out should not be a problem at all. Not one single > person, though, complained until now, since the C++ doc is totally > equivalent. > > Take a look at http://getpot.sourceforge.net for new features of the > 1.0 Version. Except for the automated documenting feature (which may > be a little intrusive in its format) I don't really see what Optik > brings new on the table. > > Guys, even though GetPot is designed for many languages, I'd like > to join the command line parsing discussion. Too late. We've settled on Optik (under a different name). --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@alum.mit.edu Thu Oct 3 16:35:20 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Thu, 3 Oct 2002 11:35:20 -0400 Subject: [Python-Dev] RFC: Option Parsing Libraries In-Reply-To: References: Message-ID: <15772.25528.921232.130849@slothrop.zope.com> >>>>> "FS" == Frank S writes: FS> After releasing the GetPot 1.0 Version I did some research on FS> other packages and I found the the Option Parsing RFC. I totally FS> missed the discussion. How come that nobody contacted me about FS> the GetPot package ? How come you didn't contact us about the GetPot package? The discussion in February began when Greg Ward proposed Optik as a standard Python module. Guido suggested that there be a general call on the python lists: If there were competing options packages, would people let us know about them. I had never heard of GetPot until I received your mail. (And if I had heard of it, I would have guessed it was illegal except for medical purposes .) So: There was general consensus that Optik should be added to the standard library. Is there a reason to include GetPot instead of Optik? Procedural issue: The (L)GPL license is unacceptable for standard library modules. I'm sure there are technical issues. Jeremy From mal@lemburg.com Thu Oct 3 16:27:10 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 03 Oct 2002 17:27:10 +0200 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <20021002221421.GA29958@thyrsus.com> <200210030022.g930Mrl06112@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3D9C61CE.9010909@lemburg.com> Guido van Rossum wrote: >>But those feelings are no proof of anything. How do we get the >>confirmation that using rationals in Python would be easy going and >>innocuous in practice, beforehand? > > > By adding them to the language but as an isolated type. The right > conversions should happen when you mix rationals with other types of > numbers (int/long -> rational -> float -> complex), but no operations > should return rationals unless a rational goes in. +1 Note that I started to work on mxNumber to get a feeling for how well rationals et al. fit the existing world. Turns out that having separate types is a goog thing. Here's the coercion scheme I'm using: mx.Number.Float ^ | --------> Python float | ^ | | | mx.Number.Rational | ^ | | Python long --> mx.Number.Integer ^ ^ | | -------- Python integer -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Thu Oct 3 16:36:08 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 03 Oct 2002 11:36:08 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Thu, 03 Oct 2002 17:27:10 +0200." <3D9C61CE.9010909@lemburg.com> References: <20021002221421.GA29958@thyrsus.com> <200210030022.g930Mrl06112@pcp02138704pcs.reston01.va.comcast.net> <3D9C61CE.9010909@lemburg.com> Message-ID: <200210031536.g93Fa8d22105@odiug.zope.com> > Note that I started to work on mxNumber to get a feeling for > how well rationals et al. fit the existing world. Turns out > that having separate types is a goo[d] thing. Any mxNumber users out there who have experience with mxRational? --Guido van Rossum (home page: http://www.python.org/~guido/) From oren-py-d@hishome.net Thu Oct 3 17:10:55 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Thu, 3 Oct 2002 19:10:55 +0300 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210031305.g93D53o12559@europa.research.att.com>; from ark@research.att.com on Thu, Oct 03, 2002 at 09:05:03AM -0400 References: <200210030216.g932G7707727@europa.research.att.com> <20021003052829.GA59708@hishome.net> <200210031305.g93D53o12559@europa.research.att.com> Message-ID: <20021003191055.A14342@hishome.net> On Thu, Oct 03, 2002 at 09:05:03AM -0400, Andrew Koenig wrote: > Oren> Any calculation involving a floating point number should taint > Oren> the result with inexactitude and yield a floating point result. > > Even comparisons? Comparisons could produce a third boolean value called 'Undetermined': result = A>B if result is Undetermined: print "Sorry, difference between A and B is below the error margin" print "I cannot determine which one is greater." else: print "%s is greater than %s" % tuple(('BA','AB')[result]) [Taking off absurdly-purist-hat :-] This actually gives me a serious idea: a floating-point type that helps keep track of error magnitude. It's a pair of floats (like a complex number) where the second number is an estimate or limit on the absolute value of errors in the first and is updated by all operators. An error tracking replacement for the math module would also be required. Oren From ark@research.att.com Thu Oct 3 17:13:33 2002 From: ark@research.att.com (Andrew Koenig) Date: Thu, 3 Oct 2002 12:13:33 -0400 (EDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021003191055.A14342@hishome.net> (message from Oren Tirosh on Thu, 3 Oct 2002 19:10:55 +0300) References: <200210030216.g932G7707727@europa.research.att.com> <20021003052829.GA59708@hishome.net> <200210031305.g93D53o12559@europa.research.att.com> <20021003191055.A14342@hishome.net> Message-ID: <200210031613.g93GDXB13582@europa.research.att.com> Oren> On Thu, Oct 03, 2002 at 09:05:03AM -0400, Andrew Koenig wrote: Oren> Any calculation involving a floating point number should taint Oren> the result with inexactitude and yield a floating point result. >> >> Even comparisons? Oren> Comparisons could produce a third boolean value called 'Undetermined': I wonder what would be the definition of sort in the presence of such comparisons? Would it be possible to implement it efficiently? From David Abrahams" <200210030216.g932G7707727@europa.research.att.com> <20021003052829.GA59708@hishome.net> <200210031305.g93D53o12559@europa.research.att.com> <20021003191055.A14342@hishome.net> <200210031613.g93GDXB13582@europa.research.att.com> Message-ID: <098501c26af5$b7d1d8b0$6701a8c0@boostconsulting.com> From: "Andrew Koenig" > Oren> On Thu, Oct 03, 2002 at 09:05:03AM -0400, Andrew Koenig wrote: > Oren> Any calculation involving a floating point number should taint > Oren> the result with inexactitude and yield a floating point result. > >> > >> Even comparisons? > > Oren> Comparisons could produce a third boolean value called 'Undetermined': It turns out that once you allow ternary logic there are too many reasonable meanings for the comparison operators. We had this long discussion around the Boost interval library. Eventually Douglas Gregor came up with a solution which seemed to satisfy everyone. I can't find it now but if anyone's interested I'll ask him to summarize the issues and solution. ----------------------------------------------------------- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From tim.one@comcast.net Thu Oct 3 17:50:38 2002 From: tim.one@comcast.net (Tim Peters) Date: Thu, 03 Oct 2002 12:50:38 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021003191055.A14342@hishome.net> Message-ID: [Oren Tirosh] > Comparisons could produce a third boolean value called 'Undetermined': Why? What real numeric problem are you trying to solve? > result = A>B > if result is Undetermined: > print "Sorry, difference between A and B is below the error margin" > print "I cannot determine which one is greater." > else: > print "%s is greater than %s" % tuple(('BA','AB')[result]) > [Taking off absurdly-purist-hat :-] > > This actually gives me a serious idea: a floating-point type that helps > keep track of error magnitude. It's a pair of floats (like a complex > number) where the second number is an estimate or limit on the absolute > value of errors in the first and is updated by all operators. It's called interval arithmetic; about 30 years ago I had to implement it for a numerical analysis class assignment, in assembler for a Univac 1108; the interface to the Fortran compiler indeed tricked it into storing the bounds in complex numbrs; *efficient* support for interval arithmetic is why the 754 std requires the directed (to-plus-inf, to-minus-inf) rounding modes; it hasn't caught on, and the current incarnation of the 754 committee is thinking about removing this requirement; in the hands of an expert it can be very powerful; in the hands of a non-expert, the usual result is that the error bounds grow so large as to render the computation useless. > An error tracking replacement for the math module would also be required. http://www.uni-karlsruhe.de/~iam/html/language/pxsc.html A single-rounding dot product proved crucial to their successes. From oren-py-d@hishome.net Thu Oct 3 17:56:02 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Thu, 3 Oct 2002 12:56:02 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210031613.g93GDXB13582@europa.research.att.com> References: <200210030216.g932G7707727@europa.research.att.com> <20021003052829.GA59708@hishome.net> <200210031305.g93D53o12559@europa.research.att.com> <20021003191055.A14342@hishome.net> <200210031613.g93GDXB13582@europa.research.att.com> Message-ID: <20021003165602.GA50630@hishome.net> On Thu, Oct 03, 2002 at 12:13:33PM -0400, Andrew Koenig wrote: > Oren> On Thu, Oct 03, 2002 at 09:05:03AM -0400, Andrew Koenig wrote: > Oren> Any calculation involving a floating point number should taint > Oren> the result with inexactitude and yield a floating point result. > >> > >> Even comparisons? > > Oren> Comparisons could produce a third boolean value called 'Undetermined': > > I wonder what would be the definition of sort in the presence > of such comparisons? Would it be possible to implement it > efficiently? That depends on the exact constraints of this lossy ordering. But let's take it even further! Let's define fuzzy booleans! an a completely Undetermined result is 0.5, meaning the comparison yielded zero information about the order of the two inputs but if their intervals of uncertainty are only partially overlapping we could get an estimate of the probability that A is greater than B. No, wait! to get better probability estimates we can define each floating point number as a CDF. This will give us information not only about the interval of uncertainty but about the actual distribution of errors. Uniform intervals are, after all, just a special case - we might want to represent numbers with gaussian errors, etc. I know it will be very useful for me in signal processing to add the gaussian noise, run the signal through a simulation of the error correction algorithm, have each undetermined comparison explore both branches in simulated parallel universes and sum the results according to their probabilities. I am a bit concerned about the performance or simple operations like addition when they require arbitrary precision convolutions of distribution functions, though. Oren From oren-py-d@hishome.net Thu Oct 3 18:43:38 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Thu, 3 Oct 2002 13:43:38 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: References: <20021003191055.A14342@hishome.net> Message-ID: <20021003174338.GB50630@hishome.net> On Thu, Oct 03, 2002 at 12:50:38PM -0400, Tim Peters wrote: > [Oren Tirosh] > > Comparisons could produce a third boolean value called 'Undetermined': > > Why? What real numeric problem are you trying to solve? I guess the smiley was camouflaged too well as the closing bracket :-) > It's called interval arithmetic; about 30 years ago I had to implement it > for a numerical analysis class assignment, in assembler for a Univac 1108; > the interface to the Fortran compiler indeed tricked it into storing the > bounds in complex numbrs; *efficient* support for interval arithmetic is why > the 754 std requires the directed (to-plus-inf, to-minus-inf) rounding > modes; it hasn't caught on, and the current incarnation of the 754 committee > is thinking about removing this requirement; in the hands of an expert it > can be very powerful; in the hands of a non-expert, the usual result is that > the error bounds grow so large as to render the computation useless. I was thinking more of estimates based on the assumption that errors are normally distributed and track standard deviation. SD grows more slowly (i.e. realistically) than true error bounds. Have you ever tried measuring noise or bit error rates in a simulation that runs thousands of times slower than real time? (change a paremeter, rinse, repeat). Something that quickly estimates the error energy could be useful. I guess that "everything is gaussian" is an approximation only an engineer could love and makes mathematicians shake their heads in disgust :-) Oren From tismer@tismer.com Thu Oct 3 19:20:54 2002 From: tismer@tismer.com (Christian Tismer) Date: Thu, 03 Oct 2002 11:20:54 -0700 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <200210030222.g932MUd20787@oma.cosc.canterbury.ac.nz> <3D9BB19D.4000401@tismer.com> <00ac01c26a89$6117aac0$b2e97ad1@othello> Message-ID: <3D9C8A86.9020805@tismer.com> Raymond Hettinger wrote: > From: "Christian Tismer" > >>>Greg> The problem as I see it is not the magnitude of the number, but >>>Greg> that the number of significant bits grows without bound, if you >>>Greg> never throw any of them away. >>> >>>So I'm suggesting that maybe it won't get bad as quickly if >>>you keep only the significant bits :-) >> >>Yes. And this is what a rounding to float would do for you. > > > That is unless you wanted more precision than comes with a float. > Normalization to some arbitrary maximum denominator magnitude > would be a more general purpose solution. Sure, this is right. So rationals are seriously considered? Would be great. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tim.one@comcast.net Thu Oct 3 19:55:46 2002 From: tim.one@comcast.net (Tim Peters) Date: Thu, 03 Oct 2002 14:55:46 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <15772.14420.986230.807066@localhost.localdomain> Message-ID: [Skip Montanaro] > ... > Is there a middle ground where you can say, in effect, "give me rational > numbers which are truncated to provide precision no worse than N-bit > floating point numbers"? Yes, so-called "fixed slash" and "floating slash" arithmetics do exactly that. BTW, they tend to give exactly the right results despite intermediate roundoff errors when the *problem* domain is such that the correct results are simple fractions. But this is just another thing that gets mentioned every time this discussion resurrects, so that's all I'm going to say about it this time around. There are literally hundreds of schemes for computer arithmetics that have been tried and found genuinely useful for *some* things. The only ones on the table with any claim to broad usefulness are rationals and decimal fp. From tismer@tismer.com Thu Oct 3 20:37:32 2002 From: tismer@tismer.com (Christian Tismer) Date: Thu, 03 Oct 2002 12:37:32 -0700 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <200210030222.g932MUd20787@oma.cosc.canterbury.ac.nz> <3D9BB19D.4000401@tismer.com> <00bf01c26adb$abf530b0$8701a8c0@holdenweb.com> Message-ID: <3D9C9C7C.7060609@tismer.com> Steve Holden wrote: [me about normalization] > Next someone will suggest that we store rationals as a sequence of > coefficients of the prime factors. This would make primes really easy to > recognise (since they'd be a sequence of zeros followed by a one with a > single one as denominator). This makes of course pretty much sense if the only operations you want are multiplication/division which becomes very very cheap, while addition becomes unaffordably expensive :-) > but-it-might-slow-down-other-operations-a-bit-ly y'rs - steve will-not-suggest-it--tried-all-of-these-decades-ago - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tim.one@comcast.net Thu Oct 3 20:53:16 2002 From: tim.one@comcast.net (Tim Peters) Date: Thu, 03 Oct 2002 15:53:16 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021003174338.GB50630@hishome.net> Message-ID: [Oren Tirosh] > ... > I was thinking more of estimates based on the assumption that errors are > normally distributed and track standard deviation. SD grows more slowly > (i.e. realistically) than true error bounds. Have you ever tried > measuring noise or bit error rates in a simulation that runs thousands > of times slower than real time? (change a paremeter, rinse, repeat). > Something that quickly estimates the error energy could be useful. If you run Emacs, get Dave Gillespie's woefully underappreciated calc-mode and look up "error forms". He did a pretty thorough implementation of the concept, including for transcendentals and complex numbers. > I guess that "everything is gaussian" is an approximation only an > engineer could love and makes mathematicians shake their heads > in disgust :-) Pure mathematicians, yes; applied mathematicians would chuckle at how cautious you are . From peter@norvig.com Thu Oct 3 21:15:01 2002 From: peter@norvig.com (Peter Norvig) Date: Thu, 03 Oct 2002 13:15:01 -0700 Subject: [Python-Dev] PEP239 (Rational Numbers) Reference Implementation and new issues References: <20021003014507.930.92987.Mailman@mail.python.org> Message-ID: <3D9CA545.8080601@norvig.com> I agree with pinard, although I think it is a manageable problem. I went through this transition once, from Symbolics Lisp, in which integer division truncates, to Common Lisp, in which it produces a rational. After only an hour or two of fiddling with my various programs, everything seemed to work; I was very happy. But after a few days, the machine started to slow down mysteriously, especially on wake-up. It took a while to discover the culprit was my screensaver program (which I hadn't checked) -- it was displaying fish on the screen, and the coordinates of the fish had become rationals quotients of thousand-digit numbers. -Peter pinard@iro.umontreal.ca writes: > While I agree with the theoretical arguments, I have the practical fear that > rationals could grow very big, rather quickly, in the course of a long > computation involving them in various ways. By big, I mean the numerator > and denominator of the fraction taken in isolation, not the number itself. From mmatus@dinha.acms.arizona.edu Thu Oct 3 22:27:48 2002 From: mmatus@dinha.acms.arizona.edu (Marcelo Matus) Date: Thu, 03 Oct 2002 14:27:48 -0700 Subject: [Python-Dev] the not operator (and the __not__ special method) Message-ID: <3D9CB654.8050306@acms.arizona.edu> Hello: The following test code: class Hello: def __not__(self): return "calling Hello.__not__" def __neg__(self): return "calling Hello.__neg__" a = Hello() print -a print not a shows that python doesn't call the __not__ special method in a 'not' operator statement. The Modules/operator.c file, line 234, says: spam2(neg,__neg__, "neg(a) -- Same as -a.") spam2(pos,__pos__, "pos(a) -- Same as +a.") spam2(abs,__abs__, "abs(a) -- Same as abs(a).") spam2(inv,__inv__, "inv(a) -- Same as ~a.") spam2(invert,__invert__, "invert(a) -- Same as ~a.") spam2(lshift,__lshift__, "lshift(a, b) -- Same as a << b.") spam2(rshift,__rshift__, "rshift(a, b) -- Same as a >> b.") spam2(not_,__not__, "not_(a) -- Same as not a.") so, one can expect that like "-a" is equivalent ot "a.__neg__()", then "not a" is equivalent to "a.__not__()", right? Another question, I notice than "a or b" and "a and b" are not equivalent to "a | b" and "a & b", since the last ones call the methods __or__ and __and__ if they are defined, but the "literal forms" never call them. Is this intentional?, if that is the case, I guess a big and clear note is needed in the documentation. Maybe the reason that the literal forms (and,or) are not equivalent to the operator forms(&,|) is preventing python to porperly resolve the "not" operation. And just for symmetry considerations, given that python has the pairs (and, &) and (or, |), does anybody considering to introduce the ! operator, so you can have the equivalent (not, !) too? Marcelo From marklists@mceahern.com Thu Oct 3 22:36:01 2002 From: marklists@mceahern.com (Mark McEahern) Date: Thu, 3 Oct 2002 16:36:01 -0500 Subject: [Python-Dev] the not operator (and the __not__ special method) In-Reply-To: <3D9CB654.8050306@acms.arizona.edu> Message-ID: http://www.python.org/doc/current/lib/module-operator.html says not_(o) __not__(o) Return the outcome of not o. (Note that there is no __not__() method for object instances; only the interpreter core defines this operation. The result is affected by the __nonzero__() and __len__() methods.) // m From guido@python.org Thu Oct 3 22:40:27 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 03 Oct 2002 17:40:27 -0400 Subject: [Python-Dev] the not operator (and the __not__ special method) In-Reply-To: Your message of "Thu, 03 Oct 2002 14:27:48 PDT." <3D9CB654.8050306@acms.arizona.edu> References: <3D9CB654.8050306@acms.arizona.edu> Message-ID: <200210032140.g93LeR930972@odiug.zope.com> > The following test code: > > > class Hello: > def __not__(self): > return "calling Hello.__not__" The reference manual (chapter 3) does not define __not__ as a special operator. > def __neg__(self): > return "calling Hello.__neg__" > > a = Hello() > print -a > print not a > > > shows that python doesn't call the __not__ special method > in a 'not' operator statement. > > The Modules/operator.c file, line 234, says: > > spam2(neg,__neg__, "neg(a) -- Same as -a.") > spam2(pos,__pos__, "pos(a) -- Same as +a.") > spam2(abs,__abs__, "abs(a) -- Same as abs(a).") > spam2(inv,__inv__, "inv(a) -- Same as ~a.") > spam2(invert,__invert__, "invert(a) -- Same as ~a.") > spam2(lshift,__lshift__, "lshift(a, b) -- Same as a << b.") > spam2(rshift,__rshift__, "rshift(a, b) -- Same as a >> b.") > spam2(not_,__not__, "not_(a) -- Same as not a.") The operator module uses __not__ as a function name for symmetry with its other function names, not because __not__ is a special operator. > so, one can expect that like "-a" is equivalent ot "a.__neg__()", > then "not a" is equivalent to "a.__not__()", right? No. > Another question, I notice than "a or b" and "a and b" are > not equivalent to "a | b" and "a & b", since the last ones call > the methods __or__ and __and__ if they are defined, but > the "literal forms" never call them. Is this intentional?, if > that is the case, I guess a big and clear note is needed > in the documentation. That all depends. The non-equivalency between 'or' and '|' (and between 'and' and '&') is quite well documented. 'or' and 'and' cannot invoke an operator, because the right operand is not evaluated at all when the left argument determines the outcome. > Maybe the reason that the literal forms (and,or) are not equivalent to > the operator forms(&,|) is preventing python to porperly > resolve the "not" operation. No. > And just for symmetry considerations, given that python has the > pairs (and, &) and (or, |), does anybody considering to introduce > the ! operator, so you can have the equivalent (not, !) too? No, there's no shortcut possibility for not as there is for 'and' and 'or'. --Guido van Rossum (home page: http://www.python.org/~guido/) From brian@sweetapp.com Thu Oct 3 22:50:54 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Thu, 03 Oct 2002 14:50:54 -0700 Subject: [Python-Dev] the not operator (and the __not__ special method) In-Reply-To: <3D9CB654.8050306@acms.arizona.edu> Message-ID: <002101c26b26$f2f1d720$df7e4e18@brianspiv1700> > shows that python doesn't call the __not__ special method > in a 'not' operator statement. Python calls the special __nonzero__ method so check the truth value of an object. > Another question, I notice than "a or b" and "a and b" are > not equivalent to "a | b" and "a & b", since the last ones call > the methods __or__ and __and__ if they are defined, but > the "literal forms" never call them. Is this intentional?, if > that is the case, I guess a big and clear note is needed > in the documentation. "and" and "or" are logical operators, while "|" and "&" are bitwise operators. >>> 13 & 14 12 >>> 13 and 14 14 > And just for symmetry considerations, given that python has the > pairs (and, &) and (or, |), does anybody considering to introduce > the ! operator, so you can have the equivalent (not, !) too? There is a bitwise not operator: ~ >>> ~2 -3 Cheers, Brian From marklists@mceahern.com Thu Oct 3 22:59:09 2002 From: marklists@mceahern.com (Mark McEahern) Date: Thu, 3 Oct 2002 16:59:09 -0500 Subject: [Python-Dev] the not operator (and the __not__ special method) In-Reply-To: <002101c26b26$f2f1d720$df7e4e18@brianspiv1700> Message-ID: > > And just for symmetry considerations, given that python has the > > pairs (and, &) and (or, |), does anybody considering to introduce > > the ! operator, so you can have the equivalent (not, !) too? > > There is a bitwise not operator: ~ > > >>> ~2 > -3 Yeah, isn't ! reserved for shouting anyway? // m From guido@python.org Thu Oct 3 23:03:45 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 03 Oct 2002 18:03:45 -0400 Subject: [Python-Dev] perplexed by mro In-Reply-To: Your message of "Tue, 01 Oct 2002 21:59:00 +0200." <01e101c26984$fc43da60$6d94fea9@newmexico> References: <01e101c26984$fc43da60$6d94fea9@newmexico> Message-ID: <200210032203.g93M3jr31166@odiug.zope.com> > I'm trying to wrap my head around the mro computation in 2.2. Thanks for doing this! You've stumbled upon a place where I'm no expert, and where my intuition about the equivalence between algorithms turned out to be wrong. > First issue: PEP 253 and the descrintro document describe an > algorithm, typeobject.c implements a rather different > one. Descrintro mentions that. > > The algorithm described by descrintro, PEP253: > > "Using the classic lookup rule, construct the list of classes that > would be searched, including duplicates. Now for each class that > occurs in the list multiple times, remove all occurrences except for > the last." > > which, if I understand correctly, means : do a left-right dfs of the > class hierarchy and then discard all but the last occurence of each > class. > > Now descrintro says that the result of this and the real algorithm > in typeobject are the same, unless > > "two given base classes occur in a different order in the > inheritance list of two different derived classes, and those derived > classes are both inherited by yet another class" > > But here is an example where the criterium does not apply but still > the algorithms' results diverge: > > >>> cpl.ex5(globals()) > class a(object): pass > class b(object): pass > class c(object): pass > class x(a): pass > class y(a): pass > class z(x,b,y,c): pass > >>> cpl.names(z.__mro__) # just the class names, real 2.2 mro algo > ['z', 'x', 'y', 'a', 'b', 'c', 'object'] > >>> cpl.names(cpl.keeplast(cpl.lrdfs(z))) # descrintro simple algo > ['z', 'x', 'b', 'y', 'a', 'c', 'object'] I wanted to say that the naive algorithm is "better" than the 2.2 algo here because it doesn't reverse the order of b and y given by z's bases. But let's have a look at an example. Suppose there's a method m that's defined in object, and overridden by class y as well as in b. What is z.m? The 2.2 algo says y.m, the naive algorithm says b.m. Hard to decide which is better. Now let's assume m is *also* overridden in a, so that y.m overrides a.m overrides object.m. Now if we searched b before y (naive algo), it is arguably strange if the 'super' call chain went from y.m via b.m to a.m. But I'm not sure how to formalize this intuition. > I was not able to find a new rule. My intuition is that when the > result of the real mro has something to do with the dfs, it is sure > from the beginning of it not the end, so the relation of the two > algos is unclear. > > In general I have an hard time trying to predict the result of the > real mro algo. > > Does it a have a list of constraints it enforces/preserves, of the > qualities it has? The book "Putting Metaclasses To Work" that I always quote about this has the algorithm and the rules it tries to enforce. I don't have the book handy here so I can't quote it. I'm pretty sure that I implemented their merge algo correctly, EXCEPT that I don't raise an error or what they call "serious order conflicts". There are no serious order conflicts in this example though. > I have tried to compare it to the algorithms used by Dylan and CLOS > and others, described in this paper > > http://www.webcom.com/haahr/dylan/linearization-oopsla96.html > > they are described in terms such as quality criteria and constraints > they preserve. Wish I could spend the time to study and understand all that. > Let's consider this example: > > >>> cpl.ex9(globals()) > class a(object): pass > class b(object): pass > class c(object): pass > class d(object): pass > class e(object): pass > class k1(a,b,c): pass > class k2(d,b,e): pass > class k3(d,a): pass > class z(k1,k2,k3): pass > >>> cpl.names(z.__mro__) > ['z', 'k1', 'k3', 'a', 'k2', 'd', 'b', 'c', 'e', 'object'] I think this one *does* have an order conflict, and in that case all bets are off about the book's algo. If you look at k1 and k2, clearly d must come after a; yet k3 wants it after a. > the thing that leaves me most perplexed is that k3 appears before > k2. All other algortihms I know would put k2 before k3. E.g. C3 > (see paper) ... > > >>> cpl.names(cpl.c3_cpl(z)) > ['z', 'k1', 'k2', 'k3', 'd', 'a', 'b', 'c', 'e', 'object'] > > or the descrintro simple algo: > > >>> cpl.names(cpl.keeplast(cpl.lrdfs(z))) > ['z', 'k1', 'c', 'k2', 'b', 'e', 'k3', 'd', 'a', 'object'] > > This means that it does not preserve the local precedence lists > (i.e. the inheritance list), it is not monotonic* because otherwise > it would put d before a, it seems to preserve "some" aspects of the > dfs traversal a,d vs d,a but still it puts k3 before k2. > > The other hypothesis, is that in this case, the algorithm should > fail, because it cannot merge the result of mro(k1) merged with > mro(k2) with mro(k3), because the latter is inconsistent with what > obtained so far? Yes. I've so far had a "don't do that then" attitude rather than a "detect all errors" attitude here -- mostly because the order conflict detection code is hairy to write in C. > >>> l=cpl.names(k1.__mro__) > >>> l > ['k1', 'a', 'b', 'c', 'object'] > >>> r=cpl.names(k2.__mro__) > >>> cpl.conservative_merge(l,r) > ... > >>> l > ['k1', 'a', 'k2', 'd', 'b', 'c', 'e', 'object'] > > vs. > > >>> cpl.names(k3.__mro__) > ['k3', 'd', 'a', 'object'] > > a > But then this inconsistency rule is different/subtler than the one > sketched in descrintro. > > So can someone (Guido?) summarize the qualities of this rule? is > there a paper that describes it? or should I buy the book? Buy the book. Last time I looked it was on sale (used) at Amazon.com. > regards. > > * the mro of a subclass is an extension without re-ordering of the > mros of the superclasses. (I don't think your second post has anything else to which I need to respond, right?) --Guido van Rossum (home page: http://www.python.org/~guido/) From mmatus@dinha.acms.arizona.edu Thu Oct 3 23:30:34 2002 From: mmatus@dinha.acms.arizona.edu (Marcelo Matus) Date: Thu, 03 Oct 2002 15:30:34 -0700 Subject: [Python-Dev] the not operator (and the __not__ special method) References: <002101c26b26$f2f1d720$df7e4e18@brianspiv1700> Message-ID: <3D9CC50A.2090208@acms.arizona.edu> Thanks to everybody for your fast and helpful answer. Clearly I was confused with the similarity between the names 'and': __and__ and 'or' : __or__, and I was thinking they were related, but that is not the case. For now, I can implement my logical classes by using the bitwise operators '&,|,~' . I forgot to say, but what I want to do is something like x = vfncs.Identity() # a class instance b = x + 2 # another class instance cond = (sin(x) <= 0.5 ) & ( b > 0.5 ) # another class instance print x(0.3) ==> 0.3 print b(0.3) ==> 2.3 print cond(0.3) ==> 1 where x, b and cond are "virtual functions" than can be evaluated like lambda functions after they are defined. The difference is that the are build from a C++ library, and they can be passed back to the C++ code where they will be evaluated directly. Anyway, given that now python has the "rich comparison" engine, where the operator <=,>,<,>= can return anything you want (and thanks for that, because I can do what I am doing now) and more boolean capabilities are been included (I read that the boolean type is coming soon), I guess it will be also good to have the special methods for the logical operations 'and', 'or' and 'not', (something like __land__, __lor__ and __lnot__ to distinguish them from the bitwise versions __and__ and __or__). So, if you overload the proper logical methods, you can write something like cond = not (sin(x) > 0.5 ) and ( b <= 0.5 ) where 'not' and 'and', like '>' and '<=', wil return anything you want. Marcelo Brian Quinlan wrote: >>shows that python doesn't call the __not__ special method >>in a 'not' operator statement. >> >> > >Python calls the special __nonzero__ method so check the truth value of >an object. > > > >>Another question, I notice than "a or b" and "a and b" are >>not equivalent to "a | b" and "a & b", since the last ones call >>the methods __or__ and __and__ if they are defined, but >>the "literal forms" never call them. Is this intentional?, if >>that is the case, I guess a big and clear note is needed >>in the documentation. >> >> > >"and" and "or" are logical operators, while "|" and "&" are bitwise >operators. > > > >>>>13 & 14 >>>> >>>> >12 > > >>>>13 and 14 >>>> >>>> >14 > > > >>And just for symmetry considerations, given that python has the >>pairs (and, &) and (or, |), does anybody considering to introduce >>the ! operator, so you can have the equivalent (not, !) too? >> >> > >There is a bitwise not operator: ~ > > > >>>>~2 >>>> >>>> >-3 > >Cheers, >Brian > > > From barry@python.org Fri Oct 4 00:02:54 2002 From: barry@python.org (Barry A. Warsaw) Date: Thu, 3 Oct 2002 19:02:54 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <20021003174338.GB50630@hishome.net> Message-ID: <15772.52382.121534.278816@gargle.gargle.HOWL> >>>>> "TP" == Tim Peters writes: TP> If you run Emacs, get Dave Gillespie's woefully TP> underappreciated calc-mode and look up "error forms". He did TP> a pretty thorough implementation of the concept, including for TP> transcendentals and complex numbers. calc-mode comes with XEmacs btw, and I still don't know what does and doesn't come with FSF Emacs. calc-mode is (and has been for years) wonderful even to add a bunch of numbers! It's nice to hear that Tim had a hand in it. -Barry From greg@cosc.canterbury.ac.nz Fri Oct 4 00:14:30 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Fri, 04 Oct 2002 11:14:30 +1200 (NZST) Subject: [Python-Dev] perplexed by mro In-Reply-To: <200210032203.g93M3jr31166@odiug.zope.com> Message-ID: <200210032314.g93NEUL29325@oma.cosc.canterbury.ac.nz> Guido: > Now let's assume m is *also* overridden in a, so that y.m overrides > a.m overrides object.m. Now if we searched b before y (naive algo), > it is arguably strange if the 'super' call chain went from y.m via b.m > to a.m. I don't understand that. According to the diagram I just scribbled, the mro from the documented algorithm comes out as z, x, b, y, a, c, object so the super chain would be b.m, y.m, a.m, object.m. (Which is *also* wrong according to my intuition, by the way -- I would expect a super call inside b.m to go to object.m, and not depend on z's inheritance structure -- but that's a separate issue.) Anyway, why not just use the algorithm you said you were using in the docs -- i.e. traverse the inheritance hierarchy and build the mro afresh each time, rather than trying to merge the mro's of the superclasses. If it's an issue of which algorithm is the one that should be used, I vote for the documented one, because it's simple and easy to explain! If it leads to odd things happening in some cases, at least it's clear what odd things will happen and why -- as opposed to other odd things happening for mysterious reasons. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg@cosc.canterbury.ac.nz Fri Oct 4 00:17:10 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Fri, 04 Oct 2002 11:17:10 +1200 (NZST) Subject: [Python-Dev] the not operator (and the __not__ special method) In-Reply-To: <3D9CC50A.2090208@acms.arizona.edu> Message-ID: <200210032317.g93NHAr29361@oma.cosc.canterbury.ac.nz> Marcelo Matus : > I guess it will be also good to have the special methods for the > logical operations 'and', 'or' and 'not', (something like __land__, > __lor__ and __lnot__ to distinguish them from the bitwise versions > __and__ and __or__). This is not easy, because the 'and' and 'or' operators have to be capable of short-circuiting -- which is the reason they don't already have __xxx__ methods. But I have some ideas on how it could be done, which I hope to get around to trying out one day. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From aahz@pythoncraft.com Fri Oct 4 00:22:15 2002 From: aahz@pythoncraft.com (Aahz) Date: Thu, 3 Oct 2002 19:22:15 -0400 Subject: [Python-Dev] the not operator (and the __not__ special method) In-Reply-To: <002101c26b26$f2f1d720$df7e4e18@brianspiv1700> References: <3D9CB654.8050306@acms.arizona.edu> <002101c26b26$f2f1d720$df7e4e18@brianspiv1700> Message-ID: <20021003232215.GA7735@panix.com> On Thu, Oct 03, 2002, Brian Quinlan wrote: > > > shows that python doesn't call the __not__ special method > > in a 'not' operator statement. > > Python calls the special __nonzero__ method so check the truth value of > an object. Mark McEahern provided the full answer: Python uses __len__, too. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Project Vote Smart: http://www.vote-smart.org/ From mmatus@dinha.acms.arizona.edu Fri Oct 4 00:45:56 2002 From: mmatus@dinha.acms.arizona.edu (Marcelo Matus) Date: Thu, 03 Oct 2002 16:45:56 -0700 Subject: [Python-Dev] PEP 285 (Adding a bool type) extending it further Message-ID: <3D9CD6B4.2050807@acms.arizona.edu> Given my silly problems trying to write a class which overwrite logical operations like 'and', 'or' and 'not', which is not possible by now (see the discussion following http://mail.python.org/pipermail/python-dev/2002-October/029122.html ) I would like to propose to extend the PEP 285 to add boolean operations that can be overloaded as the rest of the operations in python. The operations in question refers of course to 'and', 'or' and 'not'. I guess this will involve to add a structure similar to PyNumberMethods or PySequenceMethods, but for boolean methods, something like: typedef struct { binaryfunc bl_and; binaryfunc bl_or; binaryfunc bl_not; } PyBooleanMethods; where the bl_and, bl_or and bl_not methods refer to the boolean operations 'and', 'or' and 'not' of course. This methods will be the one used to implement the corresponding operations, and they could be overloaded by special methods which names could be __land__, __lor__ and __lnot__ to distinguish them from the numeric bitwise versions __and__ and __or__. In the case that there is no logical special methods defined in the class, the bolean methods must behave as they do right now. However, if the user overload them, they could return anything, just like the "rich comparison methods" >=, <=, < and >. Therefore you can write code like this class MyClass: def __land__(self, other): ..... def __lor__(self, other): ..... def _lnot__(self, other): ...... a = MyClass(1) b = MyClass(2) c = MyClass(3) res = a >= b and not (b < c) and 'res' could be anything that in the "MyClass" makes sense. Marcelo From pinard@iro.umontreal.ca Fri Oct 4 00:48:17 2002 From: pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois?= Pinard) Date: Thu, 03 Oct 2002 19:48:17 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <15772.52382.121534.278816@gargle.gargle.HOWL> (barry@python.org's message of "Thu, 3 Oct 2002 19:02:54 -0400") References: <20021003174338.GB50630@hishome.net> <15772.52382.121534.278816@gargle.gargle.HOWL> Message-ID: [Barry A. Warsaw] > calc-mode comes with XEmacs btw, and I still don't know what does and > doesn't come with FSF Emacs. I read, a year ago or so, that Calc was going to be distributed with Emacs, and maintained with it. > calc-mode is (and has been for years) wonderful even to add a bunch of > numbers! Calc is wonderful, indeed. Together with Gnus and PSGML, and surely many other things, it makes Emacs a tool which is not easy to replace. :-) -- François Pinard http://www.iro.umontreal.ca/~pinard From pinard@iro.umontreal.ca Fri Oct 4 01:19:36 2002 From: pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois?= Pinard) Date: Thu, 03 Oct 2002 20:19:36 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> (Greg Ewing's message of "Thu, 03 Oct 2002 12:42:56 +1200 (NZST)") References: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> Message-ID: [Greg Ewing] > Maybe there should be a separate operator for rational > division? > > 1/3 --> float > 1//3 --> int > 1///3 --> rational > > (Okay, a 3-char operator is a bit verbose, but I can't > think of anything else that looks division-like just at > the moment.) I much like Emacs Calc usage of 3:2 or 1:1:2 (no embedded space) for a fraction constant, read or printed. I find this more legible and comfortable to read than 3/2 or `1 1/2'. There is a Calc setting to decide if division generates floats or fractions, but Guido is probably right in choosing that `/' should not be magical enough to produce rationals. One might presume that introducing `:' between numbers at the syntactic level has just no chance to ever work. But _maybe_ it could be more tractable at the lexical level without bringing too much confusion, and for numerical _constants_ only -- 3:2 is one number, has never been two. Rationals could be got through a cast or constructor, or by introducing rational objects into the computation, like rational constants would be. -- François Pinard http://www.iro.umontreal.ca/~pinard From guido@python.org Fri Oct 4 01:19:43 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 03 Oct 2002 20:19:43 -0400 Subject: [Python-Dev] the not operator (and the __not__ special method) In-Reply-To: Your message of "Thu, 03 Oct 2002 15:30:34 PDT." <3D9CC50A.2090208@acms.arizona.edu> References: <002101c26b26$f2f1d720$df7e4e18@brianspiv1700> <3D9CC50A.2090208@acms.arizona.edu> Message-ID: <200210040019.g940JhB08869@pcp02138704pcs.reston01.va.comcast.net> > [...] I guess it will be also good to have the special methods for > the logical operations 'and', 'or' and 'not', (something like > __land__, __lor__ and __lnot__ to distinguish them from the bitwise > versions __and__ and __or__). > > So, if you overload the proper logical methods, you can write > something like > > cond = not (sin(x) > 0.5 ) and ( b <= 0.5 ) > > where 'not' and 'and', like '>' and '<=', wil return anything you want. Sorry, that won't be possible any time soon. If I allowed these operators to be overloaded, they would lose their shortcut properties, which are the reason for their existence. --Guido van Rossum (home page: http://www.python.org/~guido/) From mmatus@dinha.acms.arizona.edu Fri Oct 4 01:25:01 2002 From: mmatus@dinha.acms.arizona.edu (Marcelo Matus) Date: Thu, 03 Oct 2002 17:25:01 -0700 Subject: [Python-Dev] Re: PEP 285 (Adding a bool type) extending it further Message-ID: <3D9CDFDD.2050807@acms.arizona.edu> To solve the problem of short-circuiting 'and' and 'or', the BooleanMethod class could be tested for non zero method components, ie, you befine the methods to null, like PyBooleanMethods { binaryfunc bl_and; binaryfunc bl_or; binaryfunc bl_not; } PyBooleanMethods; static PyBooleanMethods default_as_boolean = { 0, /* bl_and */ 0, /* bl_or */ 0, /* bl_not */ }; and overwrite them only when the __land__, __lor__ or __lnot__ are defined. Then, when you perfom the 'add' or 'or' operations, you check for the proper boolean method in the first argument only, which anyway have to be evaluated. If the boolean method exists, then you force the evaluation of the second argument and the apply the proper method. If not, you do the usual, which will be the traditional short-circuiting operation, ie: a and b will be equivalent to if a.bl_and exist: return a.bl_and(b) else return a && b (traditional form with short-circuiting) and similar for 'or'. In this way, the implementation will be fully compatible with all the previous code, allowing short-circuiting for the classes that doesn't define the boolean methods (all of them by now), but will allow you to extend the operation for classes with their own version of the logical operations. Of course this requires that if you want to use your own logical operations, both or at least the first argument in the operation must always be one of your special classes. But with proper documentations and warnings about it, I guess this alternative is better than nothing. Marcelo From jeremy@alum.mit.edu Fri Oct 4 01:42:58 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Thu, 3 Oct 2002 20:42:58 -0400 Subject: [Python-Dev] Re: PEP 285 (Adding a bool type) extending it further In-Reply-To: <3D9CDFDD.2050807@acms.arizona.edu> References: <3D9CDFDD.2050807@acms.arizona.edu> Message-ID: <15772.58386.655293.791830@slothrop.zope.com> Can't you just use & and | and leave it at that? Jeremy From guido@python.org Fri Oct 4 01:52:40 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 03 Oct 2002 20:52:40 -0400 Subject: [Python-Dev] PEP 285 (Adding a bool type) extending it further In-Reply-To: Your message of "Thu, 03 Oct 2002 16:45:56 PDT." <3D9CD6B4.2050807@acms.arizona.edu> References: <3D9CD6B4.2050807@acms.arizona.edu> Message-ID: <200210040052.g940qe811749@pcp02138704pcs.reston01.va.comcast.net> > Given my silly problems trying to write a class which overwrite > logical operations like 'and', 'or' and 'not', which is not possible > by now [...] I would like to propose to extend the PEP 285 to add > boolean operations that can be overloaded as the rest of the > operations in python. Sorry, PEP 285 is already accepted and implemented, and your proposal has nothing to do with a Boolean type. I've tried to explained before why this can't be done. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 4 01:55:13 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 03 Oct 2002 20:55:13 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Thu, 03 Oct 2002 20:19:36 EDT." References: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> Message-ID: <200210040055.g940tD711784@pcp02138704pcs.reston01.va.comcast.net> > One might presume that introducing `:' between numbers at the > syntactic level has just no chance to ever work. But _maybe_ it > could be more tractable at the lexical level without bringing too > much confusion, and for numerical _constants_ only -- 3:2 is one > number, has never been two. I think the slice notation e.g. x[1:4] kills that idea. --Guido van Rossum (home page: http://www.python.org/~guido/) From mmatus@dinha.acms.arizona.edu Fri Oct 4 02:11:26 2002 From: mmatus@dinha.acms.arizona.edu (Marcelo Matus) Date: Thu, 03 Oct 2002 18:11:26 -0700 Subject: [Python-Dev] Re: PEP 285 (Adding a bool type) extending it further References: <3D9CDFDD.2050807@acms.arizona.edu> <15772.58386.655293.791830@slothrop.zope.com> Message-ID: <3D9CEABE.5020804@acms.arizona.edu> Yes, I can use the &,|,~ operators, and that is what I am doing right now. But your question also apply to the boolean type: can't you just use an integer type and leave it like that? So, my suggestion was originated because if there are users which consider relevant to have a boolean type, there will be situations when you will want to overwrite the specific 'boolean operators' too. Maybe the overwrite of the 'and' and 'or' is not the solution, maybe is better to add the && and || operators directly, like there is the pair / and //, so 'and' and 'or' will be always the old and untouched version, with the shortcut properties, and && and || could be the one you can overwrite, but they will not warranty the shortcutting. I guess that could be a happy solution for all, then a and b is the old and traditional version, and a && b could means something like: if a.bl_and exist: return a.bl_and(b) else if b.bl_rand exist: return b.bl_rand(a) else return a and b Anyway, if python allow you to do all this beatifull things overwriting operators for all the other types, and don't see anything special about the boolean type that forbid you to want to overwrite and extend its operators. Also note that the 'and' and 'or' operation have an specific meaning in the mathematical abstract sense, like '&' and '|', and the shortcut property doesn't have anynothing to do with it. In the computational implementation of them the shortcut property is usefull, because usually you apply these operators to entities which are not boolean objects and you extend their meaning. But like the operators '&' and '|' don't require the shorcut porperty, boolean types and boolean algebra don't require it either. If it is there, good, if not, good also. So, after writting this email, I would like to change the proposed extension to the inclusion of the "pure boolean" operator &&, || and !, which can be overwritten but in any way they warranty the shortcut property, like the bitwise cousins !,&,~, and leave the 'and', 'or' and 'not' operations untouched with their "extended computational meanings". Marcelo Jeremy Hylton wrote: >Can't you just use & and | and leave it at that? > >Jeremy > > > From tim.one@comcast.net Fri Oct 4 02:12:01 2002 From: tim.one@comcast.net (Tim Peters) Date: Thu, 03 Oct 2002 21:12:01 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210040055.g940tD711784@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido] > I think the slice notation e.g. x[1:4] kills that idea. Well, at least {'a':'b', 1:2:3:4:5:6} is unambiguous . not-to-mention-attractive-ly y'rs - tim From greg@cosc.canterbury.ac.nz Fri Oct 4 02:22:38 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Fri, 04 Oct 2002 13:22:38 +1200 (NZST) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210040055.g940tD711784@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210040122.g941McN29809@oma.cosc.canterbury.ac.nz> Guido: > I think the slice notation e.g. x[1:4] kills that idea. Maybe if ":" were a sliceobject-creating operator usable anwyere, not just in [...], and you added arithmetic methods to sliceobjects so you could use them as rationals... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tdelaney@avaya.com Fri Oct 4 02:50:55 2002 From: tdelaney@avaya.com (Delaney, Timothy) Date: Fri, 4 Oct 2002 11:50:55 +1000 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implemen tation and new issues Message-ID: > From: Greg Ewing [mailto:greg@cosc.canterbury.ac.nz] > > > I think the slice notation e.g. x[1:4] kills that idea. > > Maybe if ":" were a sliceobject-creating operator usable > anwyere, not just in [...], and you added arithmetic > methods to sliceobjects so you could use them as > rationals... The sad thing is this is actually tempting to me, given the beauty of using the "ratio" notation ``x:y`` to denote rational literals ... Actually, following through my thought processes, ``x:y`` is only legal in the context of a sequence slice i.e. inside ``[]``. Therefore ``x:y`` *could* be used outside of a sequence context for rationals. There would be no ambiguity if rationals could not be used in such a context, just as floats can't be (which would be consistent). e.g. ``a[1:2]`` is *always* a slice. ``b:2`` is always a rational. The problem of course is that the ``:`` operator then becomes overloaded depending on its context. Not a good thing. Damn. Just realised of course that a rational could be a dictionary index. Perhaps in that case slice takes precedence, meaning that to use a rational expression as an index it would need to be disambiguated as: ``d[(1:2)]`` This boils down to 2 rules: 1. In an indexing context, ``:`` denotes a slice expression; 2. In a non-indexing context, ``:`` denotes a rational expression. with the corollary: 1. In an indexing context, a rational expression must be disambiguated using parentheses. Tim Delaney From guido@python.org Fri Oct 4 02:54:51 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 03 Oct 2002 21:54:51 -0400 Subject: [Python-Dev] perplexed by mro In-Reply-To: Your message of "Fri, 04 Oct 2002 11:14:30 +1200." <200210032314.g93NEUL29325@oma.cosc.canterbury.ac.nz> References: <200210032314.g93NEUL29325@oma.cosc.canterbury.ac.nz> Message-ID: <200210040154.g941spF32473@pcp02138704pcs.reston01.va.comcast.net> > > Now let's assume m is *also* overridden in a, so that y.m overrides > > a.m overrides object.m. Now if we searched b before y (naive algo), > > it is arguably strange if the 'super' call chain went from y.m via b.m > > to a.m. > > I don't understand that. According to the diagram I just > scribbled, the mro from the documented algorithm comes out as > > z, x, b, y, a, c, object > > so the super chain would be b.m, y.m, a.m, object.m. Sorry, I misread my own scribbled diagram. That's indeed the correct order using the naive algorithm. And I now agree that this is a fine call chain. > (Which is *also* wrong according to my intuition, by the way -- > I would expect a super call inside b.m to go to object.m, > and not depend on z's inheritance structure -- but that's a > separate issue.) No, the whole point here is that the most inherited class's MRO (i.e. z) can insert things in a base class's MRO. Have a look at the explanation of the diamond diagram in http://python.org/2.2.1/descrintro.html > Anyway, why not just use the algorithm you said you were > using in the docs -- i.e. traverse the inheritance hierarchy > and build the mro afresh each time, rather than trying to > merge the mro's of the superclasses. I would still construct a linearized MRO out of it -- many places benefit from having it as a list rather than having to recurse down the inheritance lattice. But I agree that the naive algorithm seems to have some nice properties -- at the very least, it is easy to explain. :-) If Samuele agrees that the naive algorithm works better, I'll try to make it so in 2.3. > If it's an issue of which algorithm is the one that should > be used, I vote for the documented one, because it's > simple and easy to explain! If it leads to odd things > happening in some cases, at least it's clear what odd > things will happen and why -- as opposed to other odd > things happening for mysterious reasons. Yes. Here's how this happened: I had read and implemented the book's algorithm without really visualizing what it did in more complex cases. In the few cases that I cared about it worked fine. When I had to document it, I realized that the description in the book is very complicated, and I tried to come up with a simpler description. That's how I stumbled upon the naive algorithm -- at first I believed that it was really the same thing, but later I found a counter-example with an order disagreements, so I added that. Now Samuele has found another counter-example that does not involve an order disagreement. It does have a somewhat strange smell to it, but I can't pinpoint the quality of that smell, and it certainly doesn't have an order disagreement. I've updated the descrintro.html page to acknowledge the problems. --Guido van Rossum (home page: http://www.python.org/~guido/) From mmatus@dinha.acms.arizona.edu Fri Oct 4 03:29:25 2002 From: mmatus@dinha.acms.arizona.edu (Marcelo Matus) Date: Thu, 03 Oct 2002 19:29:25 -0700 Subject: [Python-Dev] PEP 285 (Adding a bool type) extending it further References: <3D9CD6B4.2050807@acms.arizona.edu> <200210040052.g940qe811749@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3D9CFD05.6050009@acms.arizona.edu> Guido van Rossum wrote: >>Given my silly problems trying to write a class which overwrite >>logical operations like 'and', 'or' and 'not', which is not possible >>by now [...] I would like to propose to extend the PEP 285 to add >>boolean operations that can be overloaded as the rest of the >>operations in python. >> >> > >Sorry, PEP 285 is already accepted and implemented, and your proposal >has nothing to do with a Boolean type. I've tried to explained before >why this can't be done. > >--Guido van Rossum (home page: http://www.python.org/~guido/) > > I am sorry if this is not the right forum for this topic, in that case please tell me. Also, I am sorry because it seems there were some crossover of mails, and I've got your answer after sending the request to extend the PEP 285. And when I ask to extend the PEP 285 I am not that literal, the proposed changes could be included later, maybe in another PEP. I said this proposal is an extension of the PEP 285 because it is related to the boolean support in python. And as you could see in one of the other emails, I understood the problem and the sugestions were well taken, so now what I am proposing is not to change or modify the 'and' or 'or' operators, because of the expected shortcut property. Instead what I am proposing now is the addition of the pure boolean operators &&, || and !, that like their bitwise versions &,| and ~, they don't require nor warranty any shortcut property, but they could be overloaded, and in the default case, they call the traditional 'and', 'or' and 'not' versions. Why I insist on this instead of only use the bitwise versions &,|, ~ and just "leave it at that"?, let me show you why : By now my classes are overloading the the &,|, ~ versions as replacement of the logical versions, but this have to be done with extreme carefull. Just a simple example show the problem: Assume I have a set of classes which overload the bitwise operation as a replacement of the logical ones, and then I write the following function def cond(a, b, c): return (~a | b) & ~c # for meaning (not a or b) and not c well, this function will work with my classes, but if I want to mix my classes with the real boolean and integer types, then what will be the result?. If you, for example, set a=1; b=1 and c=1 then print ~a | b & ~c ==> -2 (which means true) print (not a or b) and not c ==> 0 (which means false) therefore, replacing the 'and', 'or' and 'not' by '&,|,^' is clearly dangerous and at the end wrong because you can warranty the proper behaviour. Now, maybe this proposal "has nothing to do with a Boolean type" as you said, but once python starts to support the Boolean type, one could expect that this means more than just an uniform way to name and/or manage the True and False values. Real booelan operators and, as for almost all the other types in python, the real capability to overload them, is also important. Finally, as you can see, the shortcut property in the 'and', 'or' and 'not' operators is really important in the "computational sense", but to implement a real boolean algebra, like in the bitwise operations "&,|,^", that is secondary. Therefore, the addition of the pure boolean operators "&&,||,!" seems to be Salomonic solution. They do not interfere with the "computationaly oriented" 'and', 'or' and 'not', but they allow you to implement new (and correct) boolean relations between you own types. Marcelo From greg@cosc.canterbury.ac.nz Fri Oct 4 03:29:27 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Fri, 04 Oct 2002 14:29:27 +1200 (NZST) Subject: [Python-Dev] perplexed by mro In-Reply-To: <200210040154.g941spF32473@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210040229.g942TRQ00012@oma.cosc.canterbury.ac.nz> Guido: > No, the whole point here is that the most inherited class's MRO > (i.e. z) can insert things in a base class's MRO. It worries me that when I write a super call, I'll never be sure which method is going to be called, because someone might inherit me and mess with my mro. But I can see that there are cases where this behaviour is desirable. I guess it's okay as long as I have the option of explicitly specifying the base class. (You're not going to take that away, are you?) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one@comcast.net Fri Oct 4 03:34:00 2002 From: tim.one@comcast.net (Tim Peters) Date: Thu, 03 Oct 2002 22:34:00 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Message-ID: [Delaney, Timothy] > ... > Actually, following through my thought processes, ``x:y`` is only > legal in the context of a sequence slice i.e. inside ``[]``. Dicts are worse. {1:2:3} would be hopelessly ambiguous. From esr@thyrsus.com Fri Oct 4 03:35:08 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Thu, 3 Oct 2002 22:35:08 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implemen tation and new issues In-Reply-To: References: Message-ID: <20021004023508.GA8292@thyrsus.com> Delaney, Timothy : > 1. In an indexing context, ``:`` denotes a slice expression; > > 2. In a non-indexing context, ``:`` denotes a rational expression. > > with the corollary: > > 1. In an indexing context, a rational expression must be disambiguated using > parentheses. -1. That way lies madness^H^H^H^H^H^H^H^Perl. -- Eric S. Raymond From pedronis@bluewin.ch Fri Oct 4 03:39:30 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Fri, 4 Oct 2002 04:39:30 +0200 Subject: [Python-Dev] perplexed by mro References: <200210040229.g942TRQ00012@oma.cosc.canterbury.ac.nz> Message-ID: <041201c26b4f$444bc540$6d94fea9@newmexico> From: "Greg Ewing" > Guido: > > > No, the whole point here is that the most inherited class's MRO > > (i.e. z) can insert things in a base class's MRO. > > It worries me that when I write a super call, I'll > never be sure which method is going to be called, > because someone might inherit me and mess with > my mro. > it can add things yes, but here is were monotonicity property comes into play, if a mro algo is monotonic you know that however your class is subclassed the classes in its mro will be still encountered in the same order in the mros of the subclasses. Sadly the naive algorithm is not monotonic. >>> cpl.ex6(globals()) class a_boat(object): pass class b_day_boat(a_boat): pass class c_wheel_boat(a_boat): pass class d_engine_less(b_day_boat): pass class e_small_multihull(b_day_boat): pass class f_pedal_wheel_boat(d_engine_less,c_wheel_boat): pass class g_small_catamaran(e_small_multihull): pass class h_pedalo(f_pedal_wheel_boat,g_small_catamaran): pass >>> cpl.names(cpl.keeplast(cpl.lrdfs(f_pedal_wheel_boat))) ['f_pedal_wheel_boat', 'd_engine_less', 'b_day_boat', 'c_wheel_boat', 'a_boat', 'object'] the above computes the mro of f_pedal_wheel_boat according to naive algo, b_day_boat precedes c_wheel_boat, but for the subclass h_pedalo >>> cpl.names(cpl.keeplast(cpl.lrdfs(h_pedalo))) ['h_pedalo', 'f_pedal_wheel_boat', 'd_engine_less', 'c_wheel_boat', 'g_small_catamaran', 'e_small_multihull', 'b_day_boat', 'a_boat', 'object'] b_day_boat follows c_wheel_boat regards. From tdelaney@avaya.com Fri Oct 4 03:44:24 2002 From: tdelaney@avaya.com (Delaney, Timothy) Date: Fri, 4 Oct 2002 12:44:24 +1000 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implemen tation and new issues Message-ID: > From: Tim Peters [mailto:tim.one@comcast.net] > > [Delaney, Timothy] > > ... > > Actually, following through my thought processes, ``x:y`` is only > > legal in the context of a sequence slice i.e. inside ``[]``. > > Dicts are worse. {1:2:3} would be hopelessly ambiguous. {(1:2):3} {1:(2:3)} I'm trying to work out a way to get more parentheses in there without surrounding the actual digits ;) But it's another overloading that I missed ... As I said, it's really tempting. The concept is very elegant. Unfortunately, I don't think it's practical :( Too many cases where it would be necessary to disambiguate - even having one exception is probably too many. Do we have double-colons yet? a::b (damn - I've forgotten what that means mathematically ... equivalent?) Looks too much like C++ anyway ;) Tim Delaney From tdelaney@avaya.com Fri Oct 4 03:46:26 2002 From: tdelaney@avaya.com (Delaney, Timothy) Date: Fri, 4 Oct 2002 12:46:26 +1000 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implemen tation and new issues Message-ID: > From: Delaney, Timothy [mailto:tdelaney@avaya.com] > > Do we have double-colons yet? > > a::b Of course we do ... (slices). Tim Delaney From pedronis@bluewin.ch Fri Oct 4 03:53:23 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Fri, 4 Oct 2002 04:53:23 +0200 Subject: [Python-Dev] perplexed by mro References: <01e101c26984$fc43da60$6d94fea9@newmexico> <200210032203.g93M3jr31166@odiug.zope.com> Message-ID: <049201c26b51$34abea00$6d94fea9@newmexico> more on 2.2 algo: > > >>> cpl.ex5(globals()) > > class a(object): pass > > class b(object): pass > > class c(object): pass > > class x(a): pass > > class y(a): pass > > class z(x,b,y,c): pass > > >>> cpl.names(z.__mro__) # just the class names, real 2.2 mro algo > > ['z', 'x', 'y', 'a', 'b', 'c', 'object'] now b is there, but a simple substitution will change things radically: >>> cpl.ex5perturb(globals()) class a(object): pass class b(object): pass class c(object): pass class x(a): pass class y(a): pass class y_b(b,y): pass class z(x,y_b,c): pass >>> cpl.names(z.__mro__) ['z', 'x', 'y_b', 'b', 'y', 'a', 'c', 'object'] instead of inheriting directly from b and y, z inherits from y_b that inherits from both. Now b is between x and a. > > >>> cpl.ex9(globals()) > > class a(object): pass > > class b(object): pass > > class c(object): pass > > class d(object): pass > > class e(object): pass > > class k1(a,b,c): pass > > class k2(d,b,e): pass > > class k3(d,a): pass > > class z(k1,k2,k3): pass > > >>> cpl.names(z.__mro__) > > ['z', 'k1', 'k3', 'a', 'k2', 'd', 'b', 'c', 'e', 'object'] > > I think this one *does* have an order conflict, and in that case all > bets are off about the book's algo. If you look at k1 and k2, clearly > d must come after a; yet k3 wants it after a. > yes but for class z(k3,k1,k2) the conflict would go away. It's not intuitive nor stable. regards. From skip@pobox.com Fri Oct 4 04:00:37 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 3 Oct 2002 22:00:37 -0500 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210040122.g941McN29809@oma.cosc.canterbury.ac.nz> References: <200210040055.g940tD711784@pcp02138704pcs.reston01.va.comcast.net> <200210040122.g941McN29809@oma.cosc.canterbury.ac.nz> Message-ID: <15773.1109.273430.632460@localhost.localdomain> Greg> Guido: >> I think the slice notation e.g. x[1:4] kills that idea. Greg> Maybe if ":" were a sliceobject-creating operator usable anwyere, Greg> not just in [...], and you added arithmetic methods to Greg> sliceobjects so you could use them as rationals... Does x[1:4] return a slice out of x or x[rational(1,4)]? Skip From martin@v.loewis.de Fri Oct 4 06:24:40 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 04 Oct 2002 07:24:40 +0200 Subject: [Python-Dev] perplexed by mro In-Reply-To: <200210040229.g942TRQ00012@oma.cosc.canterbury.ac.nz> References: <200210040229.g942TRQ00012@oma.cosc.canterbury.ac.nz> Message-ID: Greg Ewing writes: > It worries me that when I write a super call, I'll > never be sure which method is going to be called, > because someone might inherit me and mess with > my mro. That's why it is called a "cooperative super call". Derived classes can *always* mess it up (be it methods or state), but this approach provides a pattern for the use case "invoke all of the base methods, and each of them once". There are cases where this is useful, and cases where this is not useful. In cases where it is useful, you have to trust derived classes to follow the pattern. Since the pattern is easy to follow, this should not be a problem. Regards, Martin From martin@v.loewis.de Fri Oct 4 06:56:28 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 04 Oct 2002 07:56:28 +0200 Subject: [Python-Dev] PEP 285 (Adding a bool type) extending it further In-Reply-To: <3D9CFD05.6050009@acms.arizona.edu> References: <3D9CD6B4.2050807@acms.arizona.edu> <200210040052.g940qe811749@pcp02138704pcs.reston01.va.comcast.net> <3D9CFD05.6050009@acms.arizona.edu> Message-ID: Marcelo Matus writes: > I am sorry if this is not the right forum for this topic, in that case > please tell me. The forum is right, but you really have to read carefully all responses you get to earlier messages. You are also supposed to do loads of research before speaking up. > Also, I am sorry because it seems there were some crossover of > mails, and I've got your answer after sending the request to extend > the PEP 285. And when I ask to extend the PEP 285 I am not that > literal, the proposed changes could be included later, maybe in > another PEP. That is unfortunate, indeed. However, this specific issue has been discussed many times before, so you could have known. > I said this proposal is an extension of the PEP 285 because it is > related to the boolean support in python. We understand why you said this. Please also understand why Guido said that PEP 285 is closed and done. > By now my classes are overloading the the &,|, ~ versions as > replacement of the logical versions, but this have to be done with > extreme carefull. Just a simple example show the problem: If you really insist on having such a feature, you'll have to give a better rationale. It is unclear to me why you have to override those operators in the first place. I would assume that every problem that involves boolean-ness of some object can be solved by implementing __nonzero__. > Now, maybe this proposal "has nothing to do with a Boolean type" as you > said, but once python starts to support the Boolean type, one could > expect that this > means more than just an uniform way to name and/or manage the True > and False > values. Real booelan operators and, as for almost all the other types > in python, the > real capability to overload them, is also important. Can you please break you lines equally? > Finally, as you can see, the shortcut property in the 'and', 'or' > and 'not' operators is really important in the "computational > sense", but to implement a real boolean algebra, like in the bitwise > operations "&,|,^", that is secondary. It so happens that the semantics is considered part of the Python language, and people rely on it. Applications would break if you took it away. In software development, the computational meaning of a program is important, not the algebraic one - even in functional languages, people could never settle on the issue of strictness. > Therefore, the addition of the pure boolean operators "&&,||,!" > seems to be Salomonic solution. I find that solution very ugly. It would add more line noise to the language, and confuse newcomers. Regards, Martin From martin@v.loewis.de Fri Oct 4 06:59:24 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 04 Oct 2002 07:59:24 +0200 Subject: [Python-Dev] Re: PEP 285 (Adding a bool type) extending it further In-Reply-To: <3D9CEABE.5020804@acms.arizona.edu> References: <3D9CDFDD.2050807@acms.arizona.edu> <15772.58386.655293.791830@slothrop.zope.com> <3D9CEABE.5020804@acms.arizona.edu> Message-ID: Marcelo Matus writes: > Yes, I can use the &,|,~ operators, and that is what I am doing right now. > > But your question also apply to the boolean type: > > can't you just use an integer type and leave it like that? I'm uncertain what you are trying to achieve. If you want to convince people, hoping that they will just see the light rarely works. Instead, you have to proceed in a diplomatic manner. If you feel resistance, analyse where this resistance comes from, and try to soften it by means of negotiation. Giving snippy responses probably won't help your cause. Regards, Martin From mmclay@comcast.net Fri Oct 4 08:46:47 2002 From: mmclay@comcast.net (Michael McLay) Date: Fri, 04 Oct 2002 03:46:47 -0400 Subject: [Python-Dev] Offtopic: Microsoft Altair BASIC legend likes Python Message-ID: <200210040346.47705.mmclay@comcast.net> Some good press for Python: http://www.theregister.co.uk/content/archive/18909.html Twenty six years ago the microprocessor revolution found a software catalyst - a tiny BASIC interpreter that ran in 4K of memory. You've probably heard of two of its three authors - Paul Allen and Bill Gates, who'd incorporated the company 'Micro-Soft' in Albuquerque the same year. The third man, Monte Davidoff, isn't nearly as famous. You'll search in vain for an interview on the web with Monte. So we figured we'd set that right. [...] His other passion, he tells us, is Python. "Hats off to them. It's an extremely well designed language. It's object orientated from the get-go. They've really succeeded there," he says, and commends it as the ideal teaching language. That used to be BASIC, of course. [...] Monte was wary about drawing generalizations about the [free software] process, but pointed out that where one or a small number of people was in charge - citing Linus and the kernel, or Guido von Rossum and Python, or Apache - the end result was demonstrably better. From mal@lemburg.com Fri Oct 4 08:57:06 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 04 Oct 2002 09:57:06 +0200 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> <200210040055.g940tD711784@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3D9D49D2.7040604@lemburg.com> [discussion on the syntax to create rationals] Are you sure that you really want a special syntax for this rather than just a simply constructor like rat(2,3) ? Just think of how difficult it would be to explain how the following would work (assuming that you use @ as the magical operator): 1 @ 2.3 1.5 @ 1.3 1j+3.4 @ 2 Note that intution isn't going to help here because you are missing a precision indicator. mxNumber has a constructor called FareyRational() which converts floats to rationals: FareyRational(value, maxden) Returns a Rational-object reflecting the given value and using maxden as maximum denominator Here's the algorithm: /* Farey Function This is a GNU MP implementation of the following function which Scott David Daniels posted to the Python Cookbook; http://www.activestate.com/ASPN/Python/Cookbook/Recipe/52317 : def farey(v, lim): '''Named after James Farey, an English surveyor. No error checking on args -- lim = max denominator, results are (numerator, denominator), (1,0) is infinity ''' if v < 0: n,d = farey(-v, lim) return -n,d z = lim-lim # get 0 of right type for denominator lower, upper = (z,z+1), (z+1,z) while 1: mediant = (lower[0] + upper[0]), (lower[1]+upper[1]) if v * mediant[1] > mediant[0]: if lim < mediant[1]: return upper lower = mediant elif v * mediant[1] == mediant[0]: if lim >= mediant[1]: return mediant if lower[1] < upper[1]: return lower return upper else: if lim < mediant[1]: return lower upper = mediant A nice proof of the algorithm can be found at "Cut the Knot": http://www.cut-the-knot.com/blue/Farey.html */ -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From oren-py-d@hishome.net Fri Oct 4 11:43:54 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Fri, 4 Oct 2002 06:43:54 -0400 Subject: [Python-Dev] the not operator (and the __not__ special method) In-Reply-To: <20021003232215.GA7735@panix.com> References: <3D9CB654.8050306@acms.arizona.edu> <002101c26b26$f2f1d720$df7e4e18@brianspiv1700> <20021003232215.GA7735@panix.com> Message-ID: <20021004104354.GA2137@hishome.net> On Thu, Oct 03, 2002 at 07:22:15PM -0400, Aahz wrote: > On Thu, Oct 03, 2002, Brian Quinlan wrote: > > > > > shows that python doesn't call the __not__ special method > > > in a 'not' operator statement. > > > > Python calls the special __nonzero__ method so check the truth value of > > an object. > > Mark McEahern provided the full answer: Python uses __len__, too. This reminds me of an asymmetry between the bool type and other built-in types: the lack of a __bool__ method. Perhaps a __bool__ method should be added to objects, just like __int__, __str__, __long__, __float__, __complex__ and __unicode__. If __bool__ is not defined bool() would fall back to using __nonzero__ and __len__, just like str() falls back to using __repr__ when __str__ is not defined. Would this be done in CPython with a new tp_bool slot or special-cased like __unicode__ and __complex__? Binary compatibility aside, I think it should be a slot. Boolean evaluation is a pretty basic operation. Oren From mwh@python.net Fri Oct 4 11:52:24 2002 From: mwh@python.net (Michael Hudson) Date: 04 Oct 2002 11:52:24 +0100 Subject: [Python-Dev] the not operator (and the __not__ special method) In-Reply-To: Oren Tirosh's message of "Fri, 4 Oct 2002 06:43:54 -0400" References: <3D9CB654.8050306@acms.arizona.edu> <002101c26b26$f2f1d720$df7e4e18@brianspiv1700> <20021003232215.GA7735@panix.com> <20021004104354.GA2137@hishome.net> Message-ID: <2md6qqbhnb.fsf@starship.python.net> Oren Tirosh writes: > On Thu, Oct 03, 2002 at 07:22:15PM -0400, Aahz wrote: > > On Thu, Oct 03, 2002, Brian Quinlan wrote: > > > > > > > shows that python doesn't call the __not__ special method > > > > in a 'not' operator statement. > > > > > > Python calls the special __nonzero__ method so check the truth value of > > > an object. > > > > Mark McEahern provided the full answer: Python uses __len__, too. > > This reminds me of an asymmetry between the bool type and other built-in > types: the lack of a __bool__ method. > > Perhaps a __bool__ method should be added to objects, just like __int__, > __str__, __long__, __float__, __complex__ and __unicode__. > > If __bool__ is not defined bool() would fall back to using __nonzero__ > and __len__, just like str() falls back to using __repr__ when __str__ is > not defined. > > Would this be done in CPython with a new tp_bool slot or special-cased > like __unicode__ and __complex__? Binary compatibility aside, I think it > should be a slot. Boolean evaluation is a pretty basic operation. I'd say just pretend "nonzero" is spelled "bool". (So "__nonzero__" == "__bool__", "tp_as_number->nb_nonzero" == "tp_bool"). It would be nice to have done this differently from the start, but it seems to me to be too much hassle to change now. Cheers, M. -- QNX... the OS that walks like a duck, quacks like a duck, but is, in fact, a platypus. ... the adventures of porting duck software to the platypus were avoidable this time. -- Chris Klein, alt.sysadmin.recovery From pinard@iro.umontreal.ca Fri Oct 4 13:06:30 2002 From: pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois?= Pinard) Date: Fri, 04 Oct 2002 08:06:30 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210040055.g940tD711784@pcp02138704pcs.reston01.va.comcast.net> (Guido van Rossum's message of "Thu, 03 Oct 2002 20:55:13 -0400") References: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> <200210040055.g940tD711784@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido van Rossum] >> But _maybe_ it could be more tractable at the lexical level without >> bringing too much confusion, and for numerical _constants_ only -- 3:2 is >> one number, has never been two. > > I think the slice notation e.g. x[1:4] kills that idea. And it kills it dead indeed, at least, in my opinion. [M.-A. Lemburg] > Are you sure that you really want a special syntax for this > rather than just a simply constructor like rat(2,3) ? This is what I currently do whenever I need rationals. It might not be as elegant as the `:' would have been, but it works well in practice. I guess I would prefer ^rat(2, 3)" over any non-elegant or non-natural notation for rational constants. We do not necessarily ought to have a special notation hardwired in Python syntax. If Guido was adding complex numbers today instead of long ago, I wonder if he would allow a special notation for them, or just suggest a constructor. > Note that intution isn't going to help here because you are missing a > precision indicator. mxNumber has a constructor called FareyRational() > which converts floats to rationals: [...] Interesting, I'll save it. I use continued fraction expansion to get the "best" rational fitting a float within a tolerance, and wonder how Farey will be similar/different. Tim will surely tell us, out of his head! :-) -- François Pinard http://www.iro.umontreal.ca/~pinard From mal@lemburg.com Fri Oct 4 13:09:22 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 04 Oct 2002 14:09:22 +0200 Subject: [Python-Dev] How to add an encoding alias? References: <200209261913.g8QJDFe01575@odiug.zope.com> <3D936369.3000908@lemburg.com> <200209262003.g8QK36P01952@odiug.zope.com> <3D936AA4.1080302@lemburg.com> <200209262027.g8QKRlO02176@odiug.zope.com> Message-ID: <3D9D84F2.4090501@lemburg.com> Guido van Rossum wrote: >>>Would it make sense to change the lookup function to convert *all* >>>punctuation to underscores before doing the lookup? (Then this one >>>would actually have worked...) >> >>Codecs must currently use names as defined by the search function in the >>encodings package: >> >> Codec modules must have names corresponding to standard lower-case >> encoding names with hyphens mapped to underscores, e.g. 'utf-8' is >> implemented by the module 'utf_8.py'. >> >>We could extend this to: >> >> Codec modules must have names corresponding to standard lower-case >> encoding names with all non-alphanumeric charactersmapped to >> underscores, e.g. 'utf-8' is implemented by the module 'utf_8.py' >> and 'ISO 639:1988' would be implemented as module 'iso_639_1988'. >> >>Note that the aliasing dictionary is consulted *after* >>having applied this mapping. > > > +1; +1 on backport to 2.2.2 also. > > Note that this requires some changes to the dict in aliases.py. Done. Not backported to 2.2.2, though, since this is a new feature. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Fri Oct 4 13:11:28 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 04 Oct 2002 08:11:28 -0400 Subject: [Python-Dev] perplexed by mro In-Reply-To: Your message of "Fri, 04 Oct 2002 14:29:27 +1200." <200210040229.g942TRQ00012@oma.cosc.canterbury.ac.nz> References: <200210040229.g942TRQ00012@oma.cosc.canterbury.ac.nz> Message-ID: <200210041211.g94CBSX20029@pcp02138704pcs.reston01.va.comcast.net> > > No, the whole point here is that the most inherited class's MRO > > (i.e. z) can insert things in a base class's MRO. > > It worries me that when I write a super call, I'll > never be sure which method is going to be called, > because someone might inherit me and mess with > my mro. But this is the whole *point* of the super() built-in. Read up on cooperative methods. > But I can see that there are cases where this > behaviour is desirable. I guess it's okay as long > as I have the option of explicitly specifying > the base class. (You're not going to take that > away, are you?) Not any time soon -- though I expect your fears are irrational. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 4 13:39:21 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 04 Oct 2002 08:39:21 -0400 Subject: [Python-Dev] the not operator (and the __not__ special method) In-Reply-To: Your message of "Fri, 04 Oct 2002 06:43:54 EDT." <20021004104354.GA2137@hishome.net> References: <3D9CB654.8050306@acms.arizona.edu> <002101c26b26$f2f1d720$df7e4e18@brianspiv1700> <20021003232215.GA7735@panix.com> <20021004104354.GA2137@hishome.net> Message-ID: <200210041239.g94CdMJ20907@pcp02138704pcs.reston01.va.comcast.net> > This reminds me of an asymmetry between the bool type and other > built-in types: the lack of a __bool__ method. > > Perhaps a __bool__ method should be added to objects, just like > __int__, __str__, __long__, __float__, __complex__ and __unicode__. > > If __bool__ is not defined bool() would fall back to using > __nonzero__ and __len__, just like str() falls back to using > __repr__ when __str__ is not defined. Apart from being a slightly odd name, __nonzero__ does this; I see no reason to rename it to __bool__, since that would add nothing new -- it would only be cosmetics. > Would this be done in CPython with a new tp_bool slot or > special-cased like __unicode__ and __complex__? Binary > compatibility aside, I think it should be a slot. Boolean > evaluation is a pretty basic operation. I think it's actually a bad idea to allow types to define their booleanness independent from their numeric value (for numbers) or length (for containers). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 4 13:54:20 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 04 Oct 2002 08:54:20 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Fri, 04 Oct 2002 08:06:30 EDT." References: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> <200210040055.g940tD711784@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210041254.g94CsKG21170@pcp02138704pcs.reston01.va.comcast.net> > If Guido was adding complex numbers today instead of long ago, I > wonder if he would allow a special notation for them, or just > suggest a constructor. At the time I believe there was heavy pressure from the Numeric crowd to allow a special notation. I'm not so sure if I should have given in though. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Fri Oct 4 14:49:58 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Fri, 4 Oct 2002 15:49:58 +0200 Subject: [Python-Dev] Re: my proposals about mros (was: perplexed by mro) References: <200210032314.g93NEUL29325@oma.cosc.canterbury.ac.nz> <200210040154.g941spF32473@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <037201c26bac$ede8d040$6d94fea9@newmexico> [GvR] > I would still construct a linearized MRO out of it -- many places > benefit from having it as a list rather than having to recurse down > the inheritance lattice. But I agree that the naive algorithm seems > to have some nice properties -- at the very least, it is easy to > explain. :-) > > If Samuele agrees that the naive algorithm works better, I'll try to > make it so in 2.3. > after cooking up the substitution example, I'm quite conviced that 2.2 mro is the worst of our options: it produces easely counterintuitive results, and if the conflict notion it needs is the one I think, it is the most prone to reject hierarchies or in case we ignore conflict to produce very odd results. So IMO our options are: *) use the naive algo: it is easy to explain (the algorithm) but we should live with the fact that it is not monotonic and I don't think it produces the most natural results (because it considers what is encoutered at the end of a dfs traversal, instead of what is at the beginning) *) adopt C3: (1) C3 is monotonic (it takes and merges respecting them the mros of the superclasses, but in parallel, non sequentially like the current 2.2 mro algo) (2) it respects also the inheritance lists of all superclasses; the paper I cited claims that is something intuitively expected, and I tend to agree. (3) it has another property, that basically enforces that, when this does not conflicts with (1) and (2), the dfs order is respected. How it works: one takes the mros of the direct superclasses and the inheritance list of the considered class cl, and merges them as ordered sequences, in case of a tie the mros of the classes appearing earlier in the inheritance list are favored (this is what induces (3)) The intuitive difference with the current 2.2 mro is that C3 uses (3) only in case of a tie, while the current 2.2 mro algo tries to enforce a a mix of (1) and (3) at the same time. >From the examples this does not seem not to have any evident benefit (unless we consider the placement of b in ex5 such), and makes it the most complicated of all algorithms from the point of view of understanding the outcomes. regards. From neal@metaslash.com Fri Oct 4 14:59:48 2002 From: neal@metaslash.com (Neal Norwitz) Date: Fri, 4 Oct 2002 09:59:48 -0400 Subject: [Python-Dev] tests failing in 2.2.2 Message-ID: <20021004135948.GA30482@epoch.metaslash.com> Is anyone else having problems with 2.2.2 tests? minidom & sax fail for me on Linux: test_sax test test_sax skipped -- cannot import name quoteattr test_minidom Traceback (most recent call last): File "Lib/test/test_minidom.py", line 636, in ? func() File "Lib/test/test_minidom.py", line 208, in testAAA "setAttribute() sets ownerDocument") File "Lib/test/test_minidom.py", line 22, in confirm raise Exception Exception I'm off for a long weekend. I'll be back late Monday. Neal From guido@python.org Fri Oct 4 16:00:33 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 04 Oct 2002 11:00:33 -0400 Subject: [Python-Dev] tests failing in 2.2.2 In-Reply-To: Your message of "Fri, 04 Oct 2002 09:59:48 EDT." <20021004135948.GA30482@epoch.metaslash.com> References: <20021004135948.GA30482@epoch.metaslash.com> Message-ID: <200210041500.g94F0XV25271@pcp02138704pcs.reston01.va.comcast.net> [Neal Norwitz] > Is anyone else having problems with 2.2.2 tests? > > minidom & sax fail for me on Linux: > > test_sax > test test_sax skipped -- cannot import name quoteattr > test_minidom > Traceback (most recent call last): > File "Lib/test/test_minidom.py", line 636, in ? > func() > File "Lib/test/test_minidom.py", line 208, in testAAA > "setAttribute() sets ownerDocument") > File "Lib/test/test_minidom.py", line 22, in confirm > raise Exception > Exception > > I'm off for a long weekend. I'll be back late Monday. Both tests work fine for me. I expect there's a problem with your expat installation. Maybe Fred has a suggestion? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 4 16:15:53 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 04 Oct 2002 11:15:53 -0400 Subject: [Python-Dev] Re: my proposals about mros (was: perplexed by mro) In-Reply-To: Your message of "Fri, 04 Oct 2002 15:49:58 +0200." <037201c26bac$ede8d040$6d94fea9@newmexico> References: <200210032314.g93NEUL29325@oma.cosc.canterbury.ac.nz> <200210040154.g941spF32473@pcp02138704pcs.reston01.va.comcast.net> <037201c26bac$ede8d040$6d94fea9@newmexico> Message-ID: <200210041515.g94FFrw25471@pcp02138704pcs.reston01.va.comcast.net> [Guido] > > If Samuele agrees that the naive algorithm works better, I'll try > > to make it so in 2.3. [Samuele] > after cooking up the substitution example, I'm quite conviced that > 2.2 mro is the worst of our options: it produces easely > counterintuitive results, and if the conflict notion it needs is the > one I think, it is the most prone to reject hierarchies or in case > we ignore conflict to produce very odd results. I don't disagree. > So IMO our options are: > *) use the naive algo: it is easy to explain (the algorithm) but we > should live with the fact that it is not monotonic and I don't think > it produces the most natural results (because it considers what is > encoutered at the end of a dfs traversal, instead of what is at the > beginning) Sadly, true. > *) adopt C3: > (1) C3 is monotonic (it takes and merges respecting them the mros of > the superclasses, but in parallel, non sequentially like the > current 2.2 mro algo) > (2) it respects also the inheritance lists of all superclasses; the > paper I cited claims that is something intuitively expected, and > I tend to agree. > (3) it has another property, that basically enforces that, when this > does not conflicts with (1) and (2), the dfs order is respected. > How it works: one takes the mros of the direct superclasses and the > inheritance list of the considered class cl, and merges them as > ordered sequences, in case of a tie the mros of the classes > appearing earlier in the inheritance list are favored (this is what > induces (3)) > > The intuitive difference with the current 2.2 mro is that C3 uses > (3) only in case of a tie, while the current 2.2 mro algo tries to > enforce a a mix of (1) and (3) at the same time. > > From the examples this does not seem not to have any evident benefit > (unless we consider the placement of b in ex5 such), and makes it > the most complicated of all algorithms from the point of view of > understanding the outcomes. I'll have to read the paper describing C3: http://www.webcom.com/haahr/dylan/linearization-oopsla96.html but I expect I'll have to agree. Thanks for your help! --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Fri Oct 4 17:27:02 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Fri, 4 Oct 2002 18:27:02 +0200 Subject: [Python-Dev] Re: my proposals about mros (was: perplexed by mro) References: <200210032314.g93NEUL29325@oma.cosc.canterbury.ac.nz> <200210040154.g941spF32473@pcp02138704pcs.reston01.va.comcast.net> <037201c26bac$ede8d040$6d94fea9@newmexico> <200210041515.g94FFrw25471@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <05d401c26bc2$df1f67c0$6d94fea9@newmexico> > I'll have to read the paper describing C3: > > http://www.webcom.com/haahr/dylan/linearization-oopsla96.html > >but I expect I'll have to agree. Thanks for your help! You're welcome, btw, I just discovered (reading the source and then the documentation, oh well, the last time I checked out of curisioty I could not find this info and did not think to look at the source) that Goo http://www.ai.mit.edu/~jrb/goo/index.htm by Jonathan Bachrach et al. at MIT is using C3 too http://www.ai.mit.edu/~jrb/goo/manual.43/goomanual_55.html From tim@zope.com Fri Oct 4 17:33:01 2002 From: tim@zope.com (Tim Peters) Date: Fri, 4 Oct 2002 12:33:01 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Message-ID: [Fran=E7ois Pinard] > ... > Interesting, I'll save it. I use continued fraction expansion to get > the "best" rational fitting a float within a tolerance, and wonder how > Farey will be similar/different. Tim will surely tell us, out of his > head! :-) Your wish is granted : Moshe Zadka's prototype implementation of rationals is still sitting in Python CVS nondist/sandox/rational/. Its _trim function is one I worked on with him, and uses c.f. expansion to fi= nd "the best" rational approximating a given rational, among all rationals w= ith denominator no larger than argument max_d. The Farey method is almost identical, except potentially much slower. It's almost what "the usual" continued fraction algorithm would do if you couldn't use integer divisio= n to do a whole bunch of steps in one gulp; e.g., whenever the c.f. expansi= on gets a partial quotient of N, the Farey method does N distinct steps. Ab= out "almost identical": c.f. expansion produces a sequence of rationals alternately smaller and larger than the target, each one (much) closer th= an the last. The Farey method also looks at rationals "between" those; _tri= m does too, but only at the endpoint, when max_d is between the denominator= s of two successive c.f. convergents. Moshe's package also has an _approximate function, to find "the smallest" rational within a given absolute distance of an input rational; that's probably closest to what you're doing now. _trim answers questions like "what's the best approximation to pi with denominator no greater than 6?". Neither of the adjacent convergents 3/1 and 22/7 is the correct answer to that; 19/6 is correct. Note that the c.f. expansion gets 22/7 because th= e previous two convergents were 1/0 and 3/1, the next partial quotient is 7= , and then the next convergent is 1 + 3*7 22 ------- =3D -- 0 + 1*7 7 If the partial quotient *had* been 6, it would have given 19/6 instead. That's what the tail end of _trim deduces. The Farey method does this one step at a time, going from (and skipping t= o then end of process) 1/0 3/1 as bounds to 3/1 4/1 and then 3/1 7/2 and then 3/1 10/3 and then 3/1 13/4 and then 3/1 16/5 and then, finally 3/1 19/6 Especially when coded in Python, it's much more efficient to deduce this = in one gulp (via exploiting division). From guido@python.org Fri Oct 4 17:52:01 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 04 Oct 2002 12:52:01 -0400 Subject: [Python-Dev] Re: my proposals about mros (was: perplexed by mro) In-Reply-To: Your message of "Fri, 04 Oct 2002 18:27:02 +0200." <05d401c26bc2$df1f67c0$6d94fea9@newmexico> References: <200210032314.g93NEUL29325@oma.cosc.canterbury.ac.nz> <200210040154.g941spF32473@pcp02138704pcs.reston01.va.comcast.net> <037201c26bac$ede8d040$6d94fea9@newmexico> <200210041515.g94FFrw25471@pcp02138704pcs.reston01.va.comcast.net> <05d401c26bc2$df1f67c0$6d94fea9@newmexico> Message-ID: <200210041652.g94Gq1F27764@pcp02138704pcs.reston01.va.comcast.net> > btw, I just discovered (reading the source and then the > documentation, oh well, the last time I checked out of curisioty I > could not find this info and did not think to look at the source) > that Goo > > http://www.ai.mit.edu/~jrb/goo/index.htm > > by Jonathan Bachrach et al. at MIT is using C3 too > > http://www.ai.mit.edu/~jrb/goo/manual.43/goomanual_55.html Could someone translate that last page into Python for me? --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Fri Oct 4 17:51:59 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Fri, 4 Oct 2002 18:51:59 +0200 Subject: [Python-Dev] Re: my proposals about mros (was: perplexed by mro) References: <200210032314.g93NEUL29325@oma.cosc.canterbury.ac.nz> <200210040154.g941spF32473@pcp02138704pcs.reston01.va.comcast.net> <037201c26bac$ede8d040$6d94fea9@newmexico> <200210041515.g94FFrw25471@pcp02138704pcs.reston01.va.comcast.net> <05d401c26bc2$df1f67c0$6d94fea9@newmexico> <200210041652.g94Gq1F27764@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <067401c26bc6$5b4d89a0$6d94fea9@newmexico> From: "Guido van Rossum" > > > Could someone translate that last page into Python for me? I can , during my exploration I have coded all known (to me) algorithms (the mysterious cpl module). A bit of polishing and I will post it. regards. From mal@lemburg.com Fri Oct 4 18:04:48 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 04 Oct 2002 19:04:48 +0200 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: Message-ID: <3D9DCA30.6010809@lemburg.com> Tim Peters wrote: > [Fran=E7ois Pinard] >=20 >>... >>Interesting, I'll save it. I use continued fraction expansion to get >>the "best" rational fitting a float within a tolerance, and wonder how >>Farey will be similar/different. Tim will surely tell us, out of his >>head! :-) >=20 >=20 > Your wish is granted : Moshe Zadka's prototype implementation of > rationals is still sitting in Python CVS nondist/sandox/rational/. Its > _trim function is one I worked on with him, and uses c.f. expansion to = find > "the best" rational approximating a given rational, among all rationals= with > denominator no larger than argument max_d. The Farey method is almost > identical, except potentially much slower. It's almost what "the usual= " > continued fraction algorithm would do if you couldn't use integer divis= ion > to do a whole bunch of steps in one gulp; e.g., whenever the c.f. expan= sion > gets a partial quotient of N, the Farey method does N distinct steps.=20 But isn't division much more costly than addition and multiplication if you have long integers to deal with ? (I can't tell, because the works are done by GMP in mxNumber) > About > "almost identical": c.f. expansion produces a sequence of rationals > alternately smaller and larger than the target, each one (much) closer = than > the last. The Farey method also looks at rationals "between" those; _t= rim > does too, but only at the endpoint, when max_d is between the denominat= ors > of two successive c.f. convergents. >=20 > Moshe's package also has an _approximate function, to find "the smalles= t" > rational within a given absolute distance of an input rational; that's > probably closest to what you're doing now. _trim answers questions lik= e > "what's the best approximation to pi with denominator no greater than 6= ?". > Neither of the adjacent convergents 3/1 and 22/7 is the correct answer = to > that; 19/6 is correct. Note that the c.f. expansion gets 22/7 because = the > previous two convergents were 1/0 and 3/1, the next partial quotient is= 7, > and then the next convergent is >=20 > 1 + 3*7 22 > ------- =3D -- > 0 + 1*7 7 >=20 > If the partial quotient *had* been 6, it would have given 19/6 instead. > That's what the tail end of _trim deduces. >=20 > The Farey method does this one step at a time, going from (and skipping= to > then end of process) >=20 > 1/0 3/1 >=20 > as bounds to >=20 > 3/1 4/1 >=20 > and then >=20 > 3/1 7/2 >=20 > and then >=20 > 3/1 10/3 >=20 > and then >=20 > 3/1 13/4 >=20 > and then >=20 > 3/1 16/5 >=20 > and then, finally >=20 > 3/1 19/6 >=20 > Especially when coded in Python, it's much more efficient to deduce thi= s in > one gulp (via exploiting division). How useful are .trim() and .approximate() in practice ? If they are, then I could put them on the TODO list for mxNumber. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From pedronis@bluewin.ch Fri Oct 4 18:31:54 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Fri, 4 Oct 2002 19:31:54 +0200 Subject: [Python-Dev] Re: my proposals about mros (was: perplexed by mro) References: <200210032314.g93NEUL29325@oma.cosc.canterbury.ac.nz> <200210040154.g941spF32473@pcp02138704pcs.reston01.va.comcast.net> <037201c26bac$ede8d040$6d94fea9@newmexico> <200210041515.g94FFrw25471@pcp02138704pcs.reston01.va.comcast.net> <05d401c26bc2$df1f67c0$6d94fea9@newmexico> <200210041652.g94Gq1F27764@pcp02138704pcs.reston01.va.comcast.net> <067401c26bc6$5b4d89a0$6d94fea9@newmexico> Message-ID: <06ac01c26bcb$eeb704a0$6d94fea9@newmexico> [me] > A bit of polishing and I will post it. > def pmerge(seqs): res = [] while 1: for seq in seqs: if seq: break else: # all sequences are empty return res cands = [] for seq in seqs: # find merge candidates among seq heads if seq: cand = seq[0] for seq in seqs: if cand in seq[1:]: break else: # cand is not in any sequences' tail => constraint-free cands.append(cand) if not cands: raise "inconsistent precedences" next = cands[0] # append next to result and remove it from sequences res.append(next) for seq in seqs: if seq and seq[0] == next: del seq[0] def c3_cpl(cl): # compute class precedence list (mro) of class cl according to C3 inputs = [[cl]] for base in cl.__bases__: cpl = c3_cpl(base) inputs.append(cpl) inputs.append(list(cl.__bases__)) # C3: considers directly also inheritance list of cl return pmerge(inputs) From tim@zope.com Fri Oct 4 18:36:59 2002 From: tim@zope.com (Tim Peters) Date: Fri, 4 Oct 2002 13:36:59 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <3D9DCA30.6010809@lemburg.com> Message-ID: [M.-A. Lemburg] > ... > But isn't division much more costly than addition and multiplication > if you have long integers to deal with ? (I can't tell, because the > works are done by GMP in mxNumber) There's no real bound on how large partial quotients can get, and rationals can grow extremely large. Dividing once to get, e.g., 10000, is enormously cheaper than going around a Python loop 10000 times, and creating and destroying several times that many temporary longs, just to avoid one relatively fast C-speed longint division with a small quotient. It's essentially the same as figuring out "the fastest" way to code a gcd, and that's a very tricky problem at the Python level. Partial quotients are *usually* 1, and then subtraction is cheaper, and it's also possible to meld both approaches to exploit that. > ... > How useful are .trim() and .approximate() in practice ? You're going to get as many responses to that as Guido got to his query about how mxNumber users like its type hierarchy . > If they are, then I could put them on the TODO list for mxNumber. They're useful for people who want to mix rationals with approximation, and that's an unlikely intersection outside of expert use. _trim is essentially what fixed-slash and floating-slash arithmetics use under the covers to keep rigorous bounds on memory use, in exchange for losing information. How useful is that in practice? Beats me; it depends so much on whose practice we're talking about . From dave@boost-consulting.com Fri Oct 4 18:36:09 2002 From: dave@boost-consulting.com (David Abrahams) Date: Fri, 4 Oct 2002 13:36:09 -0400 Subject: [Python-Dev] Python with MinGW-2.0? Message-ID: <02c601c26bcc$b1338800$6701a8c0@boostconsulting.com> Sorry if this is only mildly on-topic. If anyone here has successfully used MinGW-2.0 to build an extension module, please contact me. I'd really like to support this platform for the upcoming Boost.Python release, but it appears that MinGW-2.0 is unable to link to imported data (e.g. PyType_Type) from a stock Win32 import library such as the one that comes with the regular Windows install of Python. TIA, Dave ----------------------------------------------------------- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From haering_python@gmx.de Fri Oct 4 18:58:36 2002 From: haering_python@gmx.de (Gerhard Haering) Date: Fri, 4 Oct 2002 19:58:36 +0200 Subject: [Python-Dev] [win32] Killing MSVC's _alloca Message-ID: <20021004175835.GB95201@gargamel.ghaering.test> Trying to get around to my mingw32 port again. I currently don't have Visual C++ installed, but why is this nonstandard _alloca needed? Can't it simply be replaced by alloca? Doesn't MSVC have alloca? For the moment, I'm as far as building posixmodule.c, which I succeeded by doing a #define _alloca alloca If there's a way to kill MSVC peculiarities, could this please be done? -- Gerhard From mal@lemburg.com Fri Oct 4 19:16:57 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 04 Oct 2002 20:16:57 +0200 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: Message-ID: <3D9DDB19.3020502@lemburg.com> Tim Peters wrote: > [M.-A. Lemburg] > >>... >>But isn't division much more costly than addition and multiplication >>if you have long integers to deal with ? (I can't tell, because the >>works are done by GMP in mxNumber) > > > There's no real bound on how large partial quotients can get, and rationals > can grow extremely large. Dividing once to get, e.g., 10000, is enormously > cheaper than going around a Python loop 10000 times, and creating and > destroying several times that many temporary longs, just to avoid one > relatively fast C-speed longint division with a small quotient. Well, I'm working with GMP here, so temporary longs are not that expensive (plus they reuse already allocated memory). That's why I was asking. > It's essentially the same as figuring out "the fastest" way to code a gcd, > and that's a very tricky problem at the Python level. Partial quotients are > *usually* 1, and then subtraction is cheaper, and it's also possible to meld > both approaches to exploit that. I see. Thanks. >>... >>How useful are .trim() and .approximate() in practice ? > > > You're going to get as many responses to that as Guido got to his query > about how mxNumber users like its type hierarchy . :-) >>If they are, then I could put them on the TODO list for mxNumber. > > > They're useful for people who want to mix rationals with approximation, and > that's an unlikely intersection outside of expert use. _trim is essentially > what fixed-slash and floating-slash arithmetics use under the covers to keep > rigorous bounds on memory use, in exchange for losing information. How > useful is that in practice? Beats me; it depends so much on whose practice > we're talking about . Point taken. I just added the Farey algorithm to mxNumber because it seemed like a nice way of limiting the size of the integers involved in the rational representation of floats. It sometimes even helps to e.g. backpatch rounding/representation errors in floating point calculations when you know that your dealing with small denominator rationals: >>> 1/3.0 0.33333333333333331 >>> FareyRational(1/3.0, 100) 1/3 -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Fri Oct 4 20:31:13 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 04 Oct 2002 15:31:13 -0400 Subject: [Python-Dev] Re: my proposals about mros (was: perplexed by mro) In-Reply-To: Your message of "Fri, 04 Oct 2002 18:27:02 +0200." <05d401c26bc2$df1f67c0$6d94fea9@newmexico> References: <200210032314.g93NEUL29325@oma.cosc.canterbury.ac.nz> <200210040154.g941spF32473@pcp02138704pcs.reston01.va.comcast.net> <037201c26bac$ede8d040$6d94fea9@newmexico> <200210041515.g94FFrw25471@pcp02138704pcs.reston01.va.comcast.net> <05d401c26bc2$df1f67c0$6d94fea9@newmexico> Message-ID: <200210041931.g94JVDN29297@pcp02138704pcs.reston01.va.comcast.net> > > I'll have to read the paper describing C3: > > > > http://www.webcom.com/haahr/dylan/linearization-oopsla96.html > > > >but I expect I'll have to agree. Thanks for your help! I've read this now and I agree: we should adopt C3. I *think* that the current MRO algo is the same as the algorithm called C*[LOOPS] in that paper; for all cases where they show how C*[LOOPS] differs from Dylan or from C3, Python gives the same result as C*[LOOPS]. I wish I could find a clue in the Metaclasses book, but it's hard to search a 300-page book for an occurrence of reference [7]. :-) BTW for people coming late to this discussion, please read up on MRO (Method Resolution Order) first here: http://www.python.org/2.2.1/descrintro.html --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 4 20:40:50 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 04 Oct 2002 15:40:50 -0400 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: Your message of "Fri, 04 Oct 2002 19:58:36 +0200." <20021004175835.GB95201@gargamel.ghaering.test> References: <20021004175835.GB95201@gargamel.ghaering.test> Message-ID: <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> > Trying to get around to my mingw32 port again. I currently don't > have Visual C++ installed, but why is this nonstandard _alloca > needed? Can't it simply be replaced by alloca? Doesn't MSVC have > alloca? It seems that it does. But I guess _alloca is more politically correct, since alloca is not standard C. > For the moment, I'm as far as building posixmodule.c, which I > succeeded by doing a > > #define _alloca alloca > > If there's a way to kill MSVC peculiarities, could this please be done? I'd be happy to do a global subst of _alloca -> alloca. Mark, do you see any reason why this might *not* work? Could it break other compilers? A conservative approach would be to add #ifdef CYGWIN around the #define you propose. --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz@pythoncraft.com Fri Oct 4 20:42:48 2002 From: aahz@pythoncraft.com (Aahz) Date: Fri, 4 Oct 2002 15:42:48 -0400 Subject: [Python-Dev] Re: PEP 285 (Adding a bool type) extending it further In-Reply-To: <3D9CEABE.5020804@acms.arizona.edu> References: <3D9CDFDD.2050807@acms.arizona.edu> <15772.58386.655293.791830@slothrop.zope.com> <3D9CEABE.5020804@acms.arizona.edu> Message-ID: <20021004194248.GA28850@panix.com> On Thu, Oct 03, 2002, Marcelo Matus wrote: > > Yes, I can use the &,|,~ operators, and that is what I am doing right now. > > But your question also apply to the boolean type: > > can't you just use an integer type and leave it like that? Because Python is a delicate balancing act. We try to make it easy to learn *and* use for newcomers to programming; we try to make it easy to learn *and* use for experienced programmers; we try to keep the core language small; we try to make it powerful. Start up interactive Python and type "import this" for the extended version of this philosophy. So, specifics: Guido resisted adding a boolean type for a long time, believing that it would clutter things. Ten years of experience shows, though, that almost every medium-to-large application defines some kind of boolean type internally. Aggregate clutter is reduced by putting Booleans into the language. You want to make a base-level change like you're proposing, you need to put in the legwork to convince people that it's a net improvement. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Project Vote Smart: http://www.vote-smart.org/ From haering_python@gmx.de Fri Oct 4 21:23:48 2002 From: haering_python@gmx.de (Gerhard Haering) Date: Fri, 4 Oct 2002 22:23:48 +0200 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> References: <20021004175835.GB95201@gargamel.ghaering.test> <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021004202348.GA96145@gargamel.ghaering.test> * Guido van Rossum [2002-10-04 15:40 -0400]: > > Trying to get around to my mingw32 port again. I currently don't > > have Visual C++ installed, but why is this nonstandard _alloca > > needed? Can't it simply be replaced by alloca? Doesn't MSVC have > > alloca? > > It seems that it does. But I guess _alloca is more politically > correct, since alloca is not standard C. I don't see how this applies, as neither form is standard C, but in practise, alloca is supported by all compilers I use, be it Windows or Linux. And I don't care about P. C. ;-) > > For the moment, I'm as far as building posixmodule.c, which I > > succeeded by doing a > > > > #define _alloca alloca > > > > If there's a way to kill MSVC peculiarities, could this please be done? > > I'd be happy to do a global subst of _alloca -> alloca. > > Mark, do you see any reason why this might *not* work? > > Could it break other compilers? No, as it's only used in platform-specific code as seen below: $ find . -name *.[ch]|xargs grep -w _alloca ./Modules/posixmodule.c: s1 = (char *)_alloca(i); ./Modules/posixmodule.c: s2 = (char *)_alloca(x); ./Modules/posixmodule.c: s2 = (char *)_alloca(x); ./Python/pythonrun.c: /* _alloca throws a stack overflow exception if there's ./Python/pythonrun.c: _alloca(PYOS_STACK_MARGIN * sizeof(void*)); $ find . -name *.[ch]|xargs grep -w alloca ./Modules/mpzmodule.c:** alloca with arg < 0 (when casted to a signed ./PC/_winreg.c:#include "malloc.h" /* for alloca */ ./PC/_winreg.c: retBuf = (char *)alloca(len); ./PC/_winreg.c: retValueBuf = (char *)alloca(retValueSize); ./PC/_winreg.c: retDataBuf = (char *)alloca(retDataSize); ./PC/_winreg.c: retBuf = (char *)alloca(bufSize); ./PC/_winreg.c: retBuf = (char *)alloca(bufSize); ./PC/import_nt.c:#include "malloc.h" /* for alloca */ ./PC/import_nt.c: /* alloca == no free required, but memory only local to fn, ./PC/import_nt.c: moduleKey = alloca(bufSize); ./PC/os2vacpp/getpathp.c:#include "malloc.h" // for alloca - see comments below! ./PC/os2vacpp/getpathp.c: // alloca == no free required, but memory only local to fn. ./PC/os2vacpp/getpathp.c: keyBuf = alloca(sizeof(keyPrefix)-1 + versionLen + sizeof(keySuffix)); // chars only, plus 1 NULL. So, alloca and _alloca are only used in platform specific Windows and OS/2 code. The MSVC specific code is a little inconsistent, it uses both _alloca and alloca forms depending on the source file. > A conservative approach would be to add #ifdef CYGWIN around the > #define you propose. Not even more #ifdefs, please. I'd suggest the global replace _alloca -> alloca. -- Gerhard From martin@v.loewis.de Fri Oct 4 21:35:31 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 04 Oct 2002 22:35:31 +0200 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> References: <20021004175835.GB95201@gargamel.ghaering.test> <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > > Doesn't MSVC have alloca? > It seems that it does. Without checking: It probably has this only if __STDC__ is not defined. MSVC hides all non-standard symbols if __STDC__ is defined - either by compiler switches, or in some application header. In that sense, _alloca is the Microsoft name for this extension. They may have been misguided by using that scheme, though, since _alloca isn't any more reserved than alloca. However, all uses of _alloca in posixmodule.c are MS specific, so I really see no reason to change that. Regards, Martin From guido@python.org Fri Oct 4 21:52:30 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 04 Oct 2002 16:52:30 -0400 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: Your message of "Fri, 04 Oct 2002 22:23:48 +0200." <20021004202348.GA96145@gargamel.ghaering.test> References: <20021004175835.GB95201@gargamel.ghaering.test> <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> <20021004202348.GA96145@gargamel.ghaering.test> Message-ID: <200210042052.g94KqUj30872@pcp02138704pcs.reston01.va.comcast.net> > Not even more #ifdefs, please. I'd suggest the global replace _alloca -> > alloca. Submit a patch. --Guido van Rossum (home page: http://www.python.org/~guido/) From mhammond@skippinet.com.au Sat Oct 5 00:34:16 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Sat, 5 Oct 2002 09:34:16 +1000 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> Message-ID: > > Trying to get around to my mingw32 port again. I currently don't > > have Visual C++ installed, but why is this nonstandard _alloca > > needed? Can't it simply be replaced by alloca? Doesn't MSVC have > > alloca? > > It seems that it does. But I guess _alloca is more politically > correct, since alloca is not standard C. > > > For the moment, I'm as far as building posixmodule.c, which I > > succeeded by doing a > > > > #define _alloca alloca > > > > If there's a way to kill MSVC peculiarities, could this please be done? > > I'd be happy to do a global subst of _alloca -> alloca. > > Mark, do you see any reason why this might *not* work? None at all. Mark. From gerhard.haering@gmx.de Sat Oct 5 00:50:39 2002 From: gerhard.haering@gmx.de (Gerhard Haering) Date: Sat, 5 Oct 2002 01:50:39 +0200 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: References: <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021004235039.GA97477@gargamel.ghaering.test> * Mark Hammond [2002-10-05 09:34 +1000]: > > > [me: change _alloca to alloca] > > [Guido:] > > Mark, do you see any reason why this might *not* work? > > None at all. As per suggestion of Guido, I've submitted a patch. The patch, however, doesn't make any sense standalone, so it's part of #1 of many mingw related patches. These will ultimately make it possible to build Python on win32 using the autoconf/make toolchain. See http://python.org/sf/618791 I've assigned it to Guido for no good reason except being not sure if Mark knows about autoconf/makesetup, which my patch touches. But I'd also be glad if anybody else applies it ;-) -- Gerhard From guido@python.org Sat Oct 5 01:18:15 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 04 Oct 2002 20:18:15 -0400 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: Your message of "Sat, 05 Oct 2002 01:50:39 +0200." <20021004235039.GA97477@gargamel.ghaering.test> References: <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> <20021004235039.GA97477@gargamel.ghaering.test> Message-ID: <200210050018.g950IFO32089@pcp02138704pcs.reston01.va.comcast.net> > As per suggestion of Guido, I've submitted a patch. The patch, > however, doesn't make any sense standalone, so it's part of #1 of > many mingw related patches. So I should ignore the patch until you're done submitting those other patches? > These will ultimately make it possible > to build Python on win32 using the autoconf/make toolchain. I don't understand. How many Unix emulations on Windows do we need? We've already absorbed a sheer endless set of patches to make it work on CYGWIN. How does Mingw differ? > See http://python.org/sf/618791 > > I've assigned it to Guido for no good reason except being not sure > if Mark knows about autoconf/makesetup, which my patch touches. > > But I'd also be glad if anybody else applies it ;-) Me too. I may reduce the priority. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Sat Oct 5 02:05:53 2002 From: tim.one@comcast.net (Tim Peters) Date: Fri, 04 Oct 2002 21:05:53 -0400 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: Message-ID: [Guido] >> I'd be happy to do a global subst of _alloca -> alloca. >> >> Mark, do you see any reason why this might *not* work? [Mark] > None at all. In the interest of universal harmony, I've done this and will check it in soon (when the tests complete). no-point-arguing-over-non-issues-ly y'rs - tim From tismer@tismer.com Sat Oct 5 02:16:04 2002 From: tismer@tismer.com (Christian Tismer) Date: Fri, 04 Oct 2002 18:16:04 -0700 Subject: [Python-Dev] [win32] Killing MSVC's _alloca References: <20021004175835.GB95201@gargamel.ghaering.test> <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3D9E3D54.6020403@tismer.com> Martin v. Loewis wrote: > Guido van Rossum writes: > > >>>Doesn't MSVC have alloca? >> > >>It seems that it does. > > > Without checking: It probably has this only if __STDC__ is not > defined. MSVC hides all non-standard symbols if __STDC__ is defined - > either by compiler switches, or in some application header. Hmm. I had this problem (no alloca, no idea why) and defined it by a macro in this case. Is it cleaner to undefine __STDC__ instead? -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tim.one@comcast.net Sat Oct 5 02:23:12 2002 From: tim.one@comcast.net (Tim Peters) Date: Fri, 04 Oct 2002 21:23:12 -0400 Subject: [Python-Dev] test_pep277 vs Win98 Message-ID: The newish test_pep277 fails on Win98SE, then leaves behind a directory named @test that causes 4 later tests to fail for bogus reasons. Anyone know whether this test is supposed to work ? Mark, Martin? C:\Code\python\PCbuild>python ../lib/test/test_pep277.py test_directory (__main__.UnicodeFileTests) ... ERROR test_failures (__main__.UnicodeFileTests) ... ERROR test_listdir (__main__.UnicodeFileTests) ... ERROR test_open (__main__.UnicodeFileTests) ... ERROR test_rename (__main__.UnicodeFileTests) ... ERROR ====================================================================== ERROR: test_directory (__main__.UnicodeFileTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_pep277.py", line 31, in setUp f = open(name, 'w') IOError: [Errno 2] No such file or directory: '@test\\Ge??-sa?' ====================================================================== ERROR: test_failures (__main__.UnicodeFileTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_pep277.py", line 31, in setUp f = open(name, 'w') IOError: [Errno 2] No such file or directory: '@test\\Ge??-sa?' ====================================================================== ERROR: test_listdir (__main__.UnicodeFileTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_pep277.py", line 31, in setUp f = open(name, 'w') IOError: [Errno 2] No such file or directory: '@test\\Ge??-sa?' ====================================================================== ERROR: test_open (__main__.UnicodeFileTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_pep277.py", line 31, in setUp f = open(name, 'w') IOError: [Errno 2] No such file or directory: '@test\\Ge??-sa?' ====================================================================== ERROR: test_rename (__main__.UnicodeFileTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_pep277.py", line 31, in setUp f = open(name, 'w') IOError: [Errno 2] No such file or directory: '@test\\Ge??-sa?' ---------------------------------------------------------------------- Ran 5 tests in 0.270s FAILED (errors=5) Traceback (most recent call last): File "../lib/test/test_pep277.py", line 106, in ? test_main() File "../lib/test/test_pep277.py", line 103, in test_main run_suite(suite) File "C:\CODE\PYTHON\lib\test\test_support.py", line 187, in run_suite raise TestFailed(msg) test.test_support.TestFailed: errors occurred; run in verbose mode for details C:\Code\python\PCbuild> From gerhard.haering@gmx.de Sat Oct 5 02:26:47 2002 From: gerhard.haering@gmx.de (Gerhard Haering) Date: Sat, 5 Oct 2002 03:26:47 +0200 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: <200210050018.g950IFO32089@pcp02138704pcs.reston01.va.comcast.net> References: <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> <20021004235039.GA97477@gargamel.ghaering.test> <200210050018.g950IFO32089@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021005012647.GA98206@gargamel.ghaering.test> * Guido van Rossum [2002-10-04 20:18 -0400]: > > As per suggestion of Guido, I've submitted a patch. The patch, > > however, doesn't make any sense standalone, so it's part of #1 of > > many mingw related patches. > > So I should ignore the patch until you're done submitting those other > patches? No, this can go in now. That's the point of my plan to submit several isolated patches. I don't want to fork a Python tree, work several months on it, and then submit one gigantic patch, that will probably never be applied because there are problems reviewing it. > > These will ultimately make it possible to build Python on win32 using the > > autoconf/make toolchain. > > I don't understand. How many Unix emulations on Windows do we need? mingw is _not_ a Unix emulation. It is the GNU Compiler suite targetting _native_ Windows. The point is to be able to build Python on Windows with the autoconf toolchain. And _without_ needing to buy Visual C++ just to do build Python or to do serious Python extension development on Windows. > We've already absorbed a sheer endless set of patches to make it work on > CYGWIN. How does Mingw differ? mingw produces native win32 executables. No emulation whatsoever. > > See http://python.org/sf/618791 > > > > I've assigned it to Guido for no good reason except being not sure > > if Mark knows about autoconf/makesetup, which my patch touches. > > > > But I'd also be glad if anybody else applies it ;-) > > Me too. I may reduce the priority. The reason for setting a higher priority was that I hoped this could be applied rather sooner than later, as I hoped this wasn't a controversial patch. -- Gerhard From gerhard.haering@gmx.de Sat Oct 5 02:30:32 2002 From: gerhard.haering@gmx.de (Gerhard Haering) Date: Sat, 5 Oct 2002 03:30:32 +0200 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: <3D9E3D54.6020403@tismer.com> References: <3D9E3D54.6020403@tismer.com> Message-ID: <20021005013032.GB98206@gargamel.ghaering.test> * Christian Tismer [2002-10-04 18:16 -0700]: > Martin v. Loewis wrote: > >Guido van Rossum writes: > > > > > >>>Doesn't MSVC have alloca? > >> > > > >>It seems that it does. > > > > > >Without checking: It probably has this only if __STDC__ is not > >defined. MSVC hides all non-standard symbols if __STDC__ is defined - > >either by compiler switches, or in some application header. > > Hmm. I had this problem (no alloca, no idea why) and defined > it by a macro in this case. I've now installed MSVC6SP4 and verified that with my patch Python still builds ok. Obviously, Python's compiler options for MSVC don't define __STDC__. > Is it cleaner to undefine __STDC__ instead? It'd be cleaner to use either _alloca or alloca, but not both like the current M$VC specific code does. -- Gerhard From mhammond@skippinet.com.au Sat Oct 5 02:44:32 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Sat, 5 Oct 2002 11:44:32 +1000 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: Message-ID: > The newish test_pep277 fails on Win98SE, then leaves behind a directory > named @test that causes 4 later tests to fail for bogus reasons. > > Anyone know whether this test is supposed to work ? Mark, Martin? It is not supposed to work on Win9x Not exactly sure what to do about this. Maybe we should expose posixmodule's unicode_file_names() function - some way of indicating if the underlying file system supports Unicode. Mark. From martin@v.loewis.de Sat Oct 5 02:45:56 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 05 Oct 2002 03:45:56 +0200 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: <3D9E3D54.6020403@tismer.com> References: <20021004175835.GB95201@gargamel.ghaering.test> <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> <3D9E3D54.6020403@tismer.com> Message-ID: Christian Tismer writes: > Hmm. I had this problem (no alloca, no idea why) and defined > it by a macro in this case. > Is it cleaner to undefine __STDC__ instead? No. I would always go with the Microsoft name. If the code also needs to work on Unix, add a define for the functions you need. Regards, Martin From mhammond@skippinet.com.au Sat Oct 5 03:24:38 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Sat, 5 Oct 2002 12:24:38 +1000 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: Message-ID: > > The newish test_pep277 fails on Win98SE, then leaves behind a directory > > named @test that causes 4 later tests to fail for bogus reasons. > > > > Anyone know whether this test is supposed to work ? Mark, Martin? > > It is not supposed to work on Win9x > > Not exactly sure what to do about this. Maybe we should expose > posixmodule's unicode_file_names() function - some way of > indicating if the > underlying file system supports Unicode. OK - in the efforts of actually progressing something , would you prefer a patch that: * Added os._get_windows_version() as an alias for the Win32 GetVersion() function. We then clone the test in posixmodule.c for test_pep277.py, and life is good. This function would be generally useful - it is almost a FAQ on c.l.python and all the answers are generally unsatisfactory. * Added os._unicode_file_apis(), which exposes the internal posixmodule function unicode_file_names(). Currently this is simply a GetVersion() call, but later it may grow additional checks. * Did something else completely Mark. From martin@v.loewis.de Sat Oct 5 03:58:10 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 05 Oct 2002 04:58:10 +0200 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: References: Message-ID: Tim Peters writes: > The newish test_pep277 fails on Win98SE, then leaves behind a directory > named @test that causes 4 later tests to fail for bogus reasons. > > Anyone know whether this test is supposed to work ? Mark, Martin? To my knowledge, it can't work on Win9x. It would be good to skip that on this system. Regards, Martin From martin@v.loewis.de Sat Oct 5 04:05:58 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 05 Oct 2002 05:05:58 +0200 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: References: Message-ID: "Mark Hammond" writes: > > Not exactly sure what to do about this. Maybe we should expose > > posixmodule's unicode_file_names() function - some way of > > indicating if the underlying file system supports Unicode. I think we already rejected this approach when developing the PEP. The function name would be misleading: You can pass Unicode as a file name on all systems, thanks to the file system default encoding. > * Added os._get_windows_version() as an alias for the Win32 GetVersion() > function. We then clone the test in posixmodule.c for test_pep277.py, and > life is good. This function would be generally useful - it is almost a FAQ > on c.l.python and all the answers are generally unsatisfactory. I would like this approach, although I'm uncertain why the function name needs to start with an underscore (for that matter, I also wonder why _winreg starts with an underscore). os.windows_version() sounds good to me, although I could also accept os.get_windows_version(). Even os.version might work. > * Added os._unicode_file_apis(), which exposes the internal posixmodule > function unicode_file_names(). Currently this is simply a GetVersion() > call, but later it may grow additional checks. You like underscores really much :-) See above. Also, I think the likelyness of adding additional checks is very small: The only other candidate system to provide a similar feature would be OS X. However, macmodule.c lives elsewhere, to my knowledge. > * Did something else completely Perhaps there is already a way to find out the windows version you are running? Perhaps by looking at the registry? Regards, Martin From tim.one@comcast.net Sat Oct 5 04:09:01 2002 From: tim.one@comcast.net (Tim Peters) Date: Fri, 04 Oct 2002 23:09:01 -0400 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: Message-ID: [Mark Hammond] > OK - in the efforts of actually progressing something , would you > prefer a patch that: > > * Added os._get_windows_version() as an alias for the Win32 GetVersion() > function. We then clone the test in posixmodule.c for test_pep277.py, > and life is good. This function would be generally useful - it is > almost a FAQ on c.l.python and all the answers are generally > unsatisfactory. I agree that would be (very) useful to have, but also agree with your unstated but fiercly held belief that it's a hacky solution to the immediate problem at hand . > * Added os._unicode_file_apis(), which exposes the internal posixmodule > function unicode_file_names(). Currently this is simply a GetVersion() > call, but later it may grow additional checks. This sounds more direct, but loses me. Presumably the test is testing something that *could* be a given an informative name. Think like a Tim here: what does this test *do*? Then why is it that Windows NT+ are the only OSes on Earth where it's possible to do that thing? Or are the only OSes on Earth where it's necessary to do that thing? A Tim-friendly one-word answer to that question would give a nice name for a new boolean attribute to test: if not os.supports_FRIENDLY_NAME: rase TestSkipped, "test requires os.supports_FRIENDLY_NAME" I don't know what FRIENDLY_NAME should be, but "_unicode_file_apis()" doesn't seem likely to be it. From tim.one@comcast.net Sat Oct 5 04:11:26 2002 From: tim.one@comcast.net (Tim Peters) Date: Fri, 04 Oct 2002 23:11:26 -0400 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: Message-ID: [Martin v. Loewis] > To my knowledge, it can't work on Win9x. It would be good to skip that > on this system. Is this necessarily a different-flavors-of-Windows distinction, or is there some more fundamental underlying capability that could (and should) be given an OS-neutral name? That is, if it's not necessarily a different-flavors-of-Windows distinction for all time, testing a Windows OS version number isn't the right answer to this. From andymac@bullseye.apana.org.au Sat Oct 5 01:38:34 2002 From: andymac@bullseye.apana.org.au (Andrew MacIntyre) Date: Sat, 5 Oct 2002 11:38:34 +1100 (edt) Subject: [Python-Dev] Python with MinGW-2.0? In-Reply-To: <02c601c26bcc$b1338800$6701a8c0@boostconsulting.com> Message-ID: On Fri, 4 Oct 2002, David Abrahams wrote: > I'd really like to support this platform for the upcoming Boost.Python > release, but it appears that MinGW-2.0 is unable to link to imported data > (e.g. PyType_Type) from a stock Win32 import library such as the one that > comes with the regular Windows install of Python. My limited experience with MingW 1.1 and a DLL that contained data (and code) in the import library as well as the imports suggests that the import library has to be rebuilt to the .a format and split between static (code+data objs) and dynamic (import) libraries. This is messy & tedious to do, but seems to work (I couldn't figure out a way to combine the static objs & imports into the one .a library, though I wasn't able to spend a great deal of time on it). I gather MS "enhanced" the import library format somewhere along the line, and MingW's ld doesn't deal with it (yet). -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 andymac@pcug.org.au | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From David Abrahams" Message-ID: <011001c26c27$0c6ba220$6701a8c0@boostconsulting.com> From: "Andrew MacIntyre" > My limited experience with MingW 1.1 and a DLL that contained data (and > code) in the import library as well as the imports suggests that the > import library has to be rebuilt to the .a format and split between static > (code+data objs) and dynamic (import) libraries. This is messy & tedious > to do, but seems to work (I couldn't figure out a way to combine the > static objs & imports into the one .a library, though I wasn't able to > spend a great deal of time on it). > > I gather MS "enhanced" the import library format somewhere along the line, > and MingW's ld doesn't deal with it (yet). Thanks, For some reason I didn't have to do any of that with MinGW 1.1, but for MinGW-2.0, I did. Wasn't too painful, though: Gerhard sent a pointer to these instructions which made everything work (after substantial code massage): http://www.python.org/doc/current/inst/non-ms-compilers.html#SECTION0003120 00000000000000 ----------------------------------------------------------- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From guido@python.org Sat Oct 5 08:18:47 2002 From: guido@python.org (Guido van Rossum) Date: Sat, 05 Oct 2002 03:18:47 -0400 Subject: [Python-Dev] Python 2.2.2 beta release on Monday! Message-ID: <200210050718.g957IlF01399@pcp02138704pcs.reston01.va.comcast.net> I plan to do a Python 2.2.2 beta release this Monday (if Fred's and Tim's time is available to do their part). Please finish your changes to 2.2.2 before the weekend is over! Remember, use "cvs update -r release22-maint" to check out the 2.2.2 release branch. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Sat Oct 5 08:25:28 2002 From: tim.one@comcast.net (Tim Peters) Date: Sat, 05 Oct 2002 03:25:28 -0400 Subject: [Python-Dev] Python 2.2.2 beta release on Monday! In-Reply-To: <200210050718.g957IlF01399@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido[ > I plan to do a Python 2.2.2 beta release this Monday (if Fred's and > Tim's time is available to do their part). No problem here, and I see Fred's been working on the docs. > Please finish your changes to 2.2.2 before the weekend is over! Is there a list of outstanding issues to work on? The NEWS file is often a last-second mess . From guido@python.org Sat Oct 5 08:32:57 2002 From: guido@python.org (Guido van Rossum) Date: Sat, 05 Oct 2002 03:32:57 -0400 Subject: [Python-Dev] Python 2.2.2 beta release on Monday! In-Reply-To: Your message of "Sat, 05 Oct 2002 03:25:28 EDT." References: Message-ID: <200210050732.g957WwD01579@pcp02138704pcs.reston01.va.comcast.net> > Is there a list of outstanding issues to work on? The NEWS file is > often a last-second mess . I'm not sure it's worth mentioning every backported fix in NEWS. Backported features there aren't, except the new email package, for which Barry already added a NEWS entry. So I'm not sure there's much to do. But let's all read PEP 102 before it's too late... :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Sat Oct 5 10:26:45 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 05 Oct 2002 11:26:45 +0200 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: References: Message-ID: Tim Peters writes: > Is this necessarily a different-flavors-of-Windows distinction, or is there > some more fundamental underlying capability that could (and should) be given > an OS-neutral name? There is an abstract underlying capability "supports arbitrary Unicode file names"; the test more specifically relies on the capability "supports those 8 file names being tested". I doubt that this should be given a name at all, be it OS neutral or not. Whatever name you give it, it is likely that the capability will vary across systems, If you take a strict interpretation, you will find that no system supports it (e.g. Windows doesn't allow \ in a file name); if you take a loser definition, you find that all systems support it. > That is, if it's not necessarily a different-flavors-of-Windows > distinction for all time, testing a Windows OS version number isn't > the right answer to this. The test should be run on Win NT (and sons) only. It tests the feature that was just implemented, and that implementation is specific for Windows NT. Regards, Martin From martin@v.loewis.de Sat Oct 5 10:32:43 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 05 Oct 2002 11:32:43 +0200 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: References: Message-ID: Tim Peters writes: > Think like a Tim here: what does this test *do*? Currently, the test verifies that a certain piece of C code bracketed by #ifdef MS_WINDOWS works correctly. > Then why is it that Windows NT+ are the only OSes on Earth where > it's possible to do that thing? Testing the Windows code works can work only on Windows, by nature. It so happens that it won't work on W9x, which leaves NT+. > Or are the only OSes on Earth where it's necessary to do that thing? No, users of other systems would want that feature as well. Unfortunately, it can be correctly implemented only on OS X in addition to NT+; a somewhat-flawed implementation would be possible for Unix. > I don't know what FRIENDLY_NAME should be, but "_unicode_file_apis()" > doesn't seem likely to be it. It is really hard to find a precise name, but whatever the answer, it likely contains the words "unicode" and "file (name)". Regards, Martin From tim.one@comcast.net Sat Oct 5 19:19:11 2002 From: tim.one@comcast.net (Tim Peters) Date: Sat, 05 Oct 2002 14:19:11 -0400 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: Message-ID: [martin@v.loewis.de] > ... > There is an abstract underlying capability "supports arbitrary Unicode > file names"; the test more specifically relies on the capability > "supports those 8 file names being tested". > > I doubt that this should be given a name at all, be it OS neutral or > not. Whatever name you give it, it is likely that the capability will > vary across systems, If you take a strict interpretation, you will > find that no system supports it (e.g. Windows doesn't allow \ in a > file name); if you take a loser definition, you find that all systems > support it. > ... > It is really hard to find a precise name, but whatever the answer, it > likely contains the words "unicode" and "file (name)". This seems too much logic-chopping to me. Python supports unbounded ints too, although they're not really unbounded, and it's impossible to say exactly how unbounded they are across platforms -- or even on one platform. "Unbounded ints" remains a helpful description regardless. I'm an expert on such things, so I *could* quibble about "unbounded ints" endlessly, but it's not helpful to do so. Define os.path.supports_unicode_filenames as "supports the 8 specific filenames tested by test_pep277.py", and then that's a precisely defined lower bound that should have nothing to do with Windows specifically. After all, there is no Windows-specific code in this test! A platform supports what this test tries to do, or it doesn't. If it does, fine, then by definition it supports_unicode_filenames, and that's what the TestSkipped logic can test for. If this is thought to be a particularly stressful set of 8 specific file names (I can't guess -- "ascii" is the only one I can read ), then change the set of file names to a more reasonable one. test_long.py doesn't try to create billion-digit integers, rather it restricts itself to "unbounded ints" that any *reasonable* platform can handle. Do likewise for Unicode filename support? From martin@v.loewis.de Sat Oct 5 22:39:59 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 05 Oct 2002 23:39:59 +0200 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: References: Message-ID: Tim Peters writes: > Define os.path.supports_unicode_filenames as "supports the 8 specific > filenames tested by test_pep277.py", and then that's a precisely defined > lower bound that should have nothing to do with Windows specifically. If you take this definition, it will be very difficult to determine whether this should be True or False on Unix. On some systems, it will depend on environment variables whether these are valid file names or not. Also, the test tests whether os.listdir(u".") returns Unicode file names - which is currently never the case on Unix, even though they may support the file names as arguments. So should I add os.path.listdir_with_unicode_argument_returns_unicode as well? Furthermore, how would I implement os.path.supports_unicode_filenames for ntpath? It seems that something must be exported from os. > If this is thought to be a particularly stressful set of 8 specific > file names (I can't guess -- "ascii" is the only one I can read > ), then change the set of file names to a more reasonable one. > test_long.py doesn't try to create billion-digit integers, rather it > restricts itself to "unbounded ints" that any *reasonable* platform > can handle. Do likewise for Unicode filename support? PEP 277 gives Python applications access to all file names on Windows NT; this is a property unique to NT: on all other systems, you can access all file names using byte strings. For the test to test that feature, we need to chose a set of file names that are cannot all be represented as byte strings, simultaneously. Regards, Martin From tim.one@comcast.net Sat Oct 5 23:17:31 2002 From: tim.one@comcast.net (Tim Peters) Date: Sat, 05 Oct 2002 18:17:31 -0400 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: Message-ID: [martin@v.loewis.de] > If you take this definition, it will be very difficult to determine > whether this should be True or False on Unix. On some systems, it will > depend on environment variables whether these are valid file names or > not. Then set it to False without thinking on Unix, until Unix systems evolve to the point where Unicode strings are generally useful file paths. > Also, the test tests whether os.listdir(u".") returns Unicode file > names - which is currently never the case on Unix, even though they > may support the file names as arguments. So should I add > os.path.listdir_with_unicode_argument_returns_unicode as well? I would consider that more unhelpul logic-chopping . I would hope that intuitive listdir() behavior was part of what it *means* for Unicode strings to be generally useful as file paths. If no version of Unix supports that (yet), fine, then no version of Unix can (yet) claim supports_unicode_filenames in this sense. > Furthermore, how would I implement os.path.supports_unicode_filenames > for ntpath? It seems that something must be exported from os. This sounds like, for now, whether a platform supports this or not is a matter of hand-forcing a config symbol, where only a few Windows flavors are currently justified in saying "yup, works here". > ... > PEP 277 gives Python applications access to all file names on Windows > NT; this is a property unique to NT: on all other systems, you can > access all file names using byte strings. For the test to test that > feature, we need to chose a set of file names that are cannot all be > represented as byte strings, simultaneously. I still see nothing specific to Windows in this, apart from that NT+ appear to be, for now, the only OSes that support Unicode strings as file paths in a generally useful way. The crux of PEP 277 from my POV is not that Windows supports this, but that some flavors of Windows allow "direct access to all file names, including those that are not representable using the current locale", and using Unicode. That's a Good Idea, and I doubt Microsoft has a patent on it. The PEP also mentions that it seems *plausible* to implement this for Win9x using VFAT-32, so it's again not a "am I on Windows, and if so which flavor?" issue at heart. From martin@v.loewis.de Sun Oct 6 01:43:44 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 06 Oct 2002 02:43:44 +0200 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: References: Message-ID: Tim Peters writes: > > Furthermore, how would I implement os.path.supports_unicode_filenames > > for ntpath? It seems that something must be exported from os. > > This sounds like, for now, whether a platform supports this or not is a > matter of hand-forcing a config symbol, where only a few Windows flavors are > currently justified in saying "yup, works here". Repeating my question: How can I hand-force a config symbol to True on NT+ and to False on W9x? Regards, Martin From tim.one@comcast.net Sun Oct 6 02:05:14 2002 From: tim.one@comcast.net (Tim Peters) Date: Sat, 05 Oct 2002 21:05:14 -0400 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: Message-ID: [martin@v.loewis.de] > Repeating my question: How can I hand-force a config symbol to True on > NT+ and to False on W9x? It seemed like a rhetorical question in context , but if it's serious then offhand I'm not sure. If it's something we agree is desirable, then it needs to be a runtime decision on Windows (given that Python supports this stuff under NT+ but not (yet) under Win9x, and we only ship one Windows binary). One way to distinguish between NT+ and Win9x at runtime is in the module initialization for winsound.c. It's easy in C but exceedingly non-obvious. A more principled way would be to test for the existence of a relevant system DLL function; I expect only Mark would know whether there is such a beast in this case. From mhammond@skippinet.com.au Sun Oct 6 02:19:24 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Sun, 6 Oct 2002 11:19:24 +1000 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: Message-ID: Me: > > * Added os._get_windows_version() as an alias for the Win32 GetVersion() > > function. Martin: > I would like this approach, although I'm uncertain why the function > name needs to start with an underscore (for that matter, I also wonder > why _winreg starts with an underscore). Actually, _winreg was named that was because there was some debate over a higher level interface to the registry. There wasn't general agreement on the API, but there was agreement we could check-in a low-level "_winreg" implementation, leaving "winreg" for some presumably PEP-designed interface. > os.windows_version() sounds good to me, although I could also accept > os.get_windows_version(). Even os.version might work. And Tim later: > I agree that would be (very) useful to have, but also agree with your > unstated but fiercly held belief that it's a hacky solution to the > immediate problem at hand . Well spotted! However, pragmatism generally gets in the way . I've opted for sys.getwindowsversion() - see http://www.python.org/sf/619108 Mark. From mhammond@skippinet.com.au Sun Oct 6 02:23:30 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Sun, 6 Oct 2002 11:23:30 +1000 Subject: [Python-Dev] test_pep277 vs Win98 In-Reply-To: Message-ID: > Well spotted! However, pragmatism generally gets in the way . > I've opted for sys.getwindowsversion() - see > http://www.python.org/sf/619108 Which I created immediately before seeing the rest of this thread! Let's continue the discussion in that patch, whatever form it ends up taking. Mark. From skip@pobox.com Sun Oct 6 02:01:12 2002 From: skip@pobox.com (Skip Montanaro) Date: Sat, 5 Oct 2002 20:01:12 -0500 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210041254.g94CsKG21170@pcp02138704pcs.reston01.va.comcast.net> References: <200210030042.g930guG20522@oma.cosc.canterbury.ac.nz> <200210040055.g940tD711784@pcp02138704pcs.reston01.va.comcast.net> <200210041254.g94CsKG21170@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15775.35672.406480.187688@Skip-Montanaros-Computer.local.> >> If Guido was adding complex numbers today instead of long ago, I >> wonder if he would allow a special notation for them, or just suggest >> a constructor. Guido> At the time I believe there was heavy pressure from the Numeric Guido> crowd to allow a special notation. I'm not so sure if I should Guido> have given in though. However, 1+4j can be peephole optimized into a compile-time constant whereas complex(1,4) can't. This is generally not a big deal, but to people who deal with complex numbers a lot (and tend to be more sensitive to optimization issues) it can be. I believe when I tested my peephole optimizer using pybench several years ago, the complex number tests showed the most improvement because I could collapse constant expressions. Of course, people using lots of complex numbers probably initialize their complex constants outside of loops. ;-) Skip From tim.one@comcast.net Sun Oct 6 05:37:12 2002 From: tim.one@comcast.net (Tim Peters) Date: Sun, 06 Oct 2002 00:37:12 -0400 Subject: [Python-Dev] Testing the 2.2.2 email pkg Message-ID: Under current 2.2 CVS, running python ../lib/test/regrtest.py test_email.py from my test directory works fine. But running test_email.py directly is a disaster: python ../lib/test/test_email.py There's too much failure output to include here reasonably. Bottom line is: """ ........................FFFFFFFFFFFFFFFFFFFF...................E..FF.F. .....F..F......E......E.....E....F......................F.F............ .F.F.F............F..F ... Ran 164 tests in 0.930s FAILED (failures=33, errors=4) """ I can't yet try it in an installation, as it appears the Windows installer needs first to be taught about the new test directories under the email package (so, under an installation, test_email doesn't work at all now, no matter how it's run). From tim.one@comcast.net Sun Oct 6 06:00:07 2002 From: tim.one@comcast.net (Tim Peters) Date: Sun, 06 Oct 2002 01:00:07 -0400 Subject: [Python-Dev] 2.2 versus test_trace Message-ID: The CVS test_trace.py is failing for me when run under the debug build *and* with -O. """ C:\Code\22\PCbuild>python_d -O ../lib/test/test_trace.py Adding parser accelerators ... Done. test_1_basic (__main__.TraceTestCase) ... FAIL test_2_arigo (__main__.TraceTestCase) ... FAIL test_3_one_instr (__main__.TraceTestCase) ... FAIL test_4_no_pop_blocks (__main__.TraceTestCase) ... FAIL test_5_no_pop_tops (__main__.TraceTestCase) ... FAIL test_6_call (__main__.TraceTestCase) ... FAIL test_7_raise (__main__.TraceTestCase) ... FAIL test_8_settrace_and_return (__main__.TraceTestCase) ... FAIL test_9_settrace_and_raise (__main__.TraceTestCase) ... FAIL ====================================================================== FAIL: test_1_basic (__main__.TraceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_trace.py", line 173, in test_1_basic self.run_test(basic) File "../lib/test/test_trace.py", line 163, in run_test tracer.events, func.events) File "../lib/test/test_trace.py", line 154, in compare_events map(str, events)))) File "C:\CODE\22\lib\unittest.py", line 254, in fail raise self.failureException, msg AssertionError: events did not match expectation: (0, 'call') - (0, 'line') - (1, 'line') - (1, 'return') ? ^ + (0, 'return') ? ^ ====================================================================== FAIL: test_2_arigo (__main__.TraceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_trace.py", line 175, in test_2_arigo self.run_test(arigo_example) File "../lib/test/test_trace.py", line 163, in run_test tracer.events, func.events) File "../lib/test/test_trace.py", line 154, in compare_events map(str, events)))) File "C:\CODE\22\lib\unittest.py", line 254, in fail raise self.failureException, msg AssertionError: events did not match expectation: (0, 'call') - (0, 'line') - (1, 'line') - (2, 'line') - (3, 'line') - (3, 'line') - (5, 'line') - (5, 'return') ? ^ + (0, 'return') ? ^ ====================================================================== FAIL: test_3_one_instr (__main__.TraceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_trace.py", line 177, in test_3_one_instr self.run_test(one_instr_line) File "../lib/test/test_trace.py", line 163, in run_test tracer.events, func.events) File "../lib/test/test_trace.py", line 154, in compare_events map(str, events)))) File "C:\CODE\22\lib\unittest.py", line 254, in fail raise self.failureException, msg AssertionError: events did not match expectation: (0, 'call') - (0, 'line') - (1, 'line') - (2, 'line') - (3, 'line') - (3, 'return') ? ^ + (0, 'return') ? ^ ====================================================================== FAIL: test_4_no_pop_blocks (__main__.TraceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_trace.py", line 179, in test_4_no_pop_blocks self.run_test(no_pop_blocks) File "../lib/test/test_trace.py", line 163, in run_test tracer.events, func.events) File "../lib/test/test_trace.py", line 154, in compare_events map(str, events)))) File "C:\CODE\22\lib\unittest.py", line 254, in fail raise self.failureException, msg AssertionError: events did not match expectation: (0, 'call') - (0, 'line') - (1, 'line') - (1, 'line') - (3, 'line') - (3, 'return') ? ^ + (0, 'return') ? ^ ====================================================================== FAIL: test_5_no_pop_tops (__main__.TraceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_trace.py", line 181, in test_5_no_pop_tops self.run_test(no_pop_tops) File "../lib/test/test_trace.py", line 163, in run_test tracer.events, func.events) File "../lib/test/test_trace.py", line 154, in compare_events map(str, events)))) File "C:\CODE\22\lib\unittest.py", line 254, in fail raise self.failureException, msg AssertionError: events did not match expectation: (0, 'call') - (0, 'line') - (1, 'line') - (2, 'line') - (2, 'line') - (3, 'line') - (6, 'line') - (2, 'line') - (3, 'line') - (4, 'line') - (2, 'line') - (2, 'return') ? ^ + (0, 'return') ? ^ ====================================================================== FAIL: test_6_call (__main__.TraceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_trace.py", line 183, in test_6_call self.run_test(call) File "../lib/test/test_trace.py", line 163, in run_test tracer.events, func.events) File "../lib/test/test_trace.py", line 154, in compare_events map(str, events)))) File "C:\CODE\22\lib\unittest.py", line 254, in fail raise self.failureException, msg AssertionError: events did not match expectation: (0, 'call') - (0, 'line') - (1, 'line') (-3, 'call') - (-3, 'line') - (-2, 'line') - (-2, 'return') ? ^ + (-3, 'return') ? ^ - (1, 'return') ? ^ + (0, 'return') ? ^ ====================================================================== FAIL: test_7_raise (__main__.TraceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_trace.py", line 185, in test_7_raise self.run_test(test_raise) File "../lib/test/test_trace.py", line 163, in run_test tracer.events, func.events) File "../lib/test/test_trace.py", line 154, in compare_events map(str, events)))) File "C:\CODE\22\lib\unittest.py", line 254, in fail raise self.failureException, msg AssertionError: events did not match expectation: (0, 'call') - (0, 'line') - (1, 'line') - (2, 'line') (-3, 'call') - (-3, 'line') - (-2, 'line') - (-2, 'exception') ? ^ + (-3, 'exception') ? ^ - (2, 'exception') ? ^ + (0, 'exception') ? ^ - (3, 'line') - (4, 'line') - (4, 'return') ? ^ + (0, 'return') ? ^ ====================================================================== FAIL: test_8_settrace_and_return (__main__.TraceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_trace.py", line 188, in test_8_settrace_and_return self.run_test2(settrace_and_return) File "../lib/test/test_trace.py", line 170, in run_test2 tracer.events, func.events) File "../lib/test/test_trace.py", line 154, in compare_events map(str, events)))) File "C:\CODE\22\lib\unittest.py", line 254, in fail raise self.failureException, msg AssertionError: events did not match expectation: - (1, 'return') ? ^ + (0, 'return') ? ^ ====================================================================== FAIL: test_9_settrace_and_raise (__main__.TraceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_trace.py", line 190, in test_9_settrace_and_raise self.run_test2(settrace_and_raise) File "../lib/test/test_trace.py", line 170, in run_test2 tracer.events, func.events) File "../lib/test/test_trace.py", line 154, in compare_events map(str, events)))) File "C:\CODE\22\lib\unittest.py", line 254, in fail raise self.failureException, msg AssertionError: events did not match expectation: - (2, 'exception') ? ^ + (0, 'exception') ? ^ - (3, 'line') - (4, 'line') - (4, 'return') ? ^ + (0, 'return') ? ^ ---------------------------------------------------------------------- Ran 9 tests in 0.220s FAILED (failures=9) Traceback (most recent call last): File "../lib/test/test_trace.py", line 216, in ? test_main() File "../lib/test/test_trace.py", line 212, in test_main test_support.run_unittest(TraceTestCase) File "../lib/test\test_support.py", line 180, in run_unittest run_suite(unittest.makeSuite(testclass), testclass) File "../lib/test\test_support.py", line 174, in run_suite raise TestFailed(msg) test_support.TestFailed: errors occurred in __main__.TraceTestCase [6768 refs] C:\Code\22\PCbuild> """ It also fails under the release build with -O. It does not fail in either build if I leave -O off. This smells familiar, and I think I fixed this before. I don't want to fix it again . From skip@manatee.mojam.com Sun Oct 6 13:00:24 2002 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 6 Oct 2002 07:00:24 -0500 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200210061200.g96C0OnL027526@manatee.mojam.com> Bug/Patch Summary ----------------- 293 open / 2905 total bugs (+1) 112 open / 1719 total patches (+3) New Bugs -------- packages doc in the tutorial (2002-09-29) http://python.org/sf/616211 plat-linux2/IN.py FutureWarning's (2002-10-03) http://python.org/sf/618012 overflow error in calendar module (2002-10-03) http://python.org/sf/618146 Windows binary missing IPv6 support (2002-10-04) http://python.org/sf/618593 Connecting to IPv4 addrs from IPv6 socke (2002-10-04) http://python.org/sf/618594 list(xrange(sys.maxint/4)) again (2002-10-04) http://python.org/sf/618623 Use C3 MRO algorithm (2002-10-04) http://python.org/sf/618704 New Patches ----------- textwrap: prefix short lines (2002-09-30) http://python.org/sf/616496 getframe hook (Psyco #1) (2002-10-01) http://python.org/sf/617309 Tiny profiling info (Psyco #2) (2002-10-01) http://python.org/sf/617311 debugger-controlled jumps (Psyco #3) (2002-10-01) http://python.org/sf/617312 Rational Reference Implementation (2002-10-02) http://python.org/sf/617779 gzip.py and files > 2G (2002-10-03) http://python.org/sf/618135 [mingw patches] alloca and posixmodule (2002-10-04) http://python.org/sf/618791 Fix test_pep277 on Win9x (2002-10-05) http://python.org/sf/619108 Closed Bugs ----------- ImportError: No module named _socket (2002-06-07) http://python.org/sf/565710 NetBSD 1.4.3, a.out, shared modules (2002-08-17) http://python.org/sf/596576 socket.getfqdn() doesn't on Windows (2002-09-27) http://python.org/sf/615472 getdefaultlocale failure on OS X (2002-09-28) http://python.org/sf/616002 Closed Patches -------------- test_commands test fails under Cygwin (2002-04-16) http://python.org/sf/544740 PEP 277: Unicode file name support (2002-08-12) http://python.org/sf/594001 ccompiler argument checking too strict (2002-09-02) http://python.org/sf/603831 build fixes for SCO (2002-09-26) http://python.org/sf/615069 acconfig.h out of date (2002-09-26) http://python.org/sf/615343 From barry@wooz.org Sun Oct 6 13:17:34 2002 From: barry@wooz.org (Barry A. Warsaw) Date: Sun, 6 Oct 2002 08:17:34 -0400 Subject: [Python-Dev] Re: Testing the 2.2.2 email pkg References: Message-ID: <15776.10718.561784.209566@gargle.gargle.HOWL> >>>>> "TP" == Tim Peters writes: TP> Under current 2.2 CVS, running TP> python ../lib/test/regrtest.py test_email.py TP> from my test directory works fine. But running test_email.py TP> directly is a disaster: TP> python ../lib/test/test_email.py TP> There's too much failure output to include here reasonably. TP> Bottom line is: TP> """ TP> ........................FFFFFFFFFFFFFFFFFFFF...................E..FF.F. TP> .....F..F......E......E.....E....F......................F.F............ TP> .F.F.F............F..F TP> ... TP> Ran 164 tests in 0.930s | FAILED (failures=33, errors=4) | """ TP> I can't yet try it in an installation, as it appears the TP> Windows installer needs first to be taught about the new test TP> directories under the email package (so, under an TP> installation, test_email doesn't work at all now, no matter TP> how it's run). It's gotta be a Windows thing: % cat CVS/Tag Trelease22-maint % cvs -q up -P -d % make % ./python Lib/test/test_email.py .................................................................................................................................................................... ---------------------------------------------------------------------- Ran 164 tests in 1.179s OK This on RH7.3. Could it be the "cvs transforms line endings" problem we looked at a few weeks ago? Does Python 2.3 cvs have the same problem (it's got a nearly identical snapshot of the email package)? If you want to send me the test output directly, feel free, although I'm not sure how much time I'll have today to look at it. -Barry From guido@python.org Sun Oct 6 14:29:24 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 06 Oct 2002 09:29:24 -0400 Subject: [Python-Dev] Re: [snake-farm] Autodetect /usr/ccs/bin? In-Reply-To: Your message of "Sun, 06 Oct 2002 13:57:14 +0200." <20021006115713.GB22400@lysator.liu.se> References: <20021006115713.GB22400@lysator.liu.se> Message-ID: <200210061329.g96DTOs14718@pcp02138704pcs.reston01.va.comcast.net> [Anders Qvist] > Several machines in the snake farm fail to build libpython2.3 because > they don't have ar in their path. ar, ld and the like live in > /usr/ccs/bin, which seems to be some kind of standard (for generous > values of standard) as both SunOS 5.8, AIX 4.3 and HP/UX 11.00 seems > ot have this dir. > > Should the configure script try with /usr/ccs/bin/ar and ld if none is > explicitly supplied? This may be a good idea, but I don't know zit about this area. I'm cc'ing python-dev, maybeo someone there understands this. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sun Oct 6 15:11:33 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 06 Oct 2002 10:11:33 -0400 Subject: [Python-Dev] Re: Testing the 2.2.2 email pkg In-Reply-To: Your message of "Sun, 06 Oct 2002 08:17:34 EDT." <15776.10718.561784.209566@gargle.gargle.HOWL> References: <15776.10718.561784.209566@gargle.gargle.HOWL> Message-ID: <200210061411.g96EBXa18691@pcp02138704pcs.reston01.va.comcast.net> There's one file in the email/test/data directory that contains newlines when I check it out: msg_26.txt. I find it hard to believe that alone would cause 33 failures and 4 errors, but who knows. I note that I see a similar set of errors and failures with Python 2.3, so it appears to be a general problem with the new email package's test suite on Windows, when invoked manually (python Lib/test/test_email.py). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sun Oct 6 15:19:18 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 06 Oct 2002 10:19:18 -0400 Subject: [Python-Dev] Re: Testing the 2.2.2 email pkg In-Reply-To: Your message of "Sun, 06 Oct 2002 10:11:33 EDT." <200210061411.g96EBXa18691@pcp02138704pcs.reston01.va.comcast.net> References: <15776.10718.561784.209566@gargle.gargle.HOWL> <200210061411.g96EBXa18691@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210061419.g96EJIu18736@pcp02138704pcs.reston01.va.comcast.net> > I note that I see a similar set of errors and failures with Python > 2.3, so it appears to be a general problem with the new email > package's test suite on Windows, when invoked manually (python > Lib/test/test_email.py). Another thought just struck me. Are we certain that the email test suite is run when you use regrtest? I think not! [linux]$ time ./python ../Lib/test/test_email.py .................................................................................................................................................................... ---------------------------------------------------------------------- Ran 164 tests in 0.899s OK real 0m1.136s user 0m1.020s sys 0m0.070s [linux]$ time ./python ../Lib/test/regrtest.py test_email test_email 1 test OK. real 0m0.248s user 0m0.220s sys 0m0.030s [linux]$ The times of the second run suggest that no tests were run! So the email regression test (or the email package!) probably has a severe Unix dependency (or Windows deficiency :-) that has never been detected before. I'll mail the full test output from Windows box next. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Sun Oct 6 16:06:28 2002 From: tim.one@comcast.net (Tim Peters) Date: Sun, 06 Oct 2002 11:06:28 -0400 Subject: [Python-Dev] Re: Testing the 2.2.2 email pkg In-Reply-To: <200210061419.g96EJIu18736@pcp02138704pcs.reston01.va.comcast.net> Message-ID: This gets too confusing too fast. test_email fails under 2.3 CVS for me this morning too, no matter how I run it -- via regrtest, by itself, doesn't matter. STRAIGHT REGREST >python ../lib/test/regrtest.py test_email test_email test test_email failed -- errors occurred; run in verbose mode for details 1 test failed: test_email DITTO BUT VERBOSE >python ../lib/test/regrtest.py -v test_email ... lots of error output ... Ran 164 tests in 1.150s FAILED (failures=33, errors=4) test test_email failed -- errors occurred; run in verbose mode for details 1 test failed: test_email BY ITSELF >python ../lib/test/test_email.py ... lots of error output ... Ran 164 tests in 1.150s FAILED (failures=33, errors=4) Traceback (most recent call last): File "../lib/test/test_email.py", line 13, in ? test_main() File "../lib/test/test_email.py", line 10, in test_main run_suite(suite()) File "C:\CODE\PYTHON\lib\test\test_support.py", line 187, in run_suite raise TestFailed(msg) test.test_support.TestFailed: errors occurred; run in verbose mode for details Why I'm getting a traceback from test_support despite not using regrtest seems a bit of a mystery. SKIPPING THE TEST DIRECTORY ENTIRELY >python ../lib/email/test/test_email.py ... lots of error output ... Ran 164 tests in 0.770s FAILED (failures=33, errors=4) > Another thought just struck me. Are we certain that the email test > suite is run when you use regrtest? I think not! As above, it certainly is for me, under current 2.3 CVS. > ... > So the email regression test (or the email package!) probably has a > severe Unix dependency (or Windows deficiency :-) that has never been > detected before. Barry knows he has cross-platform line-end problems in his test data, but hasn't wanted to "do something" about that so far. I'm most unclear now on why test_email ever worked on Windows; *something* changed very recently, but don't know what.... lots of error output ... From tim.one@comcast.net Sun Oct 6 16:11:53 2002 From: tim.one@comcast.net (Tim Peters) Date: Sun, 06 Oct 2002 11:11:53 -0400 Subject: [Python-Dev] Re: Testing the 2.2.2 email pkg In-Reply-To: <200210061411.g96EBXa18691@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido] > ... > I note that I see a similar set of errors and failures with Python > 2.3, so it appears to be a general problem with the new email > package's test suite on Windows, when invoked manually (python > Lib/test/test_email.py). This morning (but not yesterday) test_email fails for me here (under 2.3 CVS) even when run indirectly, via just running regrtest. I assume some checkin occurred I haven't located yet. From tim.one@comcast.net Sun Oct 6 16:28:41 2002 From: tim.one@comcast.net (Tim Peters) Date: Sun, 06 Oct 2002 11:28:41 -0400 Subject: [Python-Dev] RE: Testing the 2.2.2 email pkg In-Reply-To: <15776.10718.561784.209566@gargle.gargle.HOWL> Message-ID: [Barry] > ... > Could it be the "cvs transforms line endings" problem we looked at a > few weeks ago? The problem we looked at a few weeks ago was that WinZip transforms line endings in text files, unless you disable that (it's a well-hidden WinZip option). CVS also transforms text-file line endings, which is something we did not look at before, but which should have been obvious at the time. I'm sure it's the same problem. It was hidden because regrtest.py didn't actually run the email tests until Guido fixed that this morning. Now test_email on Windows fails no matter how it's run. > Does Python 2.3 cvs have the same problem (it's got a nearly identical > snapshot of the email package)? Since Guido fixed regrtest-vs-test_email this morning, yes. > If you want to send me the test output directly, feel free, although > I'm not sure how much time I'll have today to look at it. You already know what the problem is, so no need. From paul@pfdubois.com Sun Oct 6 17:14:01 2002 From: paul@pfdubois.com (Paul F Dubois) Date: Sun, 6 Oct 2002 09:14:01 -0700 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021006120102.30351.31268.Mailman@mail.python.org> Message-ID: <000001c26d53$64253b70$6701a8c0@NICKLEBY> The question was raised as to why the Numeric community really wanted the literal notation 1.0+2.0j for complex numbers. The motivation was not speed particularly; nobody is going to construct a large complex array out of literals. It is true that inside a loop a constant might be a problem but you could always lift it by hand. I'd say rather that the real motivation was ease of learning for those who knew similar languages. There were essentially three ways this was done in other languages that our users would already grok: (1.0, 2.0) -- Fortran; a non-starter for Python 1.0 + 2.0i or 1.0 + 2.0j -- most (all?) existing tools for similar purposes use one of these. We all felt that Python would be perceived as "inferior" to these other programs if you had to do something clumsy. I haven't followed the rational discussion but obviously there is no other long-standing and highly used notation for rationals except 1.0/2.0, is there? From jacobs@penguin.theopalgroup.com Sun Oct 6 18:13:30 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Sun, 6 Oct 2002 13:13:30 -0400 (EDT) Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ Message-ID: Hi Guido and python-dev, Python 2.2.2 beta 1 is coming out, and I was hoping to float a final request to have the fix for bug #537450 applied to the 2.2 maint branch (it is a trivial backport, since the 2.3 patch applies just fine). A back-port was originally passed over, because it changes semantics from Python 2.2.1. I argue that the changed semantics are desirable and constitute a clear bug fix, which closes holes that can allow very obscure errors. It would be very surprising to me if this change breaks any extant code, since it is virtually impossible to take meaningful or safe advantage of the original semantics. Here is, in brief, an example of what I am talking about (the bug and python-dev archives have more information on this topic). Let's define a class Foo: class Foo(object): def __new__(cls): return [1,2,3] print Foo() Okay, this _is_ a contrived example, though it does reflects a real use-case that appears in one of my large real-world apps. Here is the output from Python 2.2.1: >>> print Foo() [] # !!! not what we expected! and from Python 2.3 CVS: >>> print Foo() [1, 2, 3] # Just right! This is because any object returned from __new__ has its constructor called, regardless of its type in 2.2.1. The new rule in Python 2.3 CVS is that the constructor is only called when the type returned is an instance of the type being built (including subtype instances). The above example demonstrates the problems with the 2.2.1 behavior on lists, though more problems are exposed when we consider other object types: class Bar(object): def __init__(self, a, b, c): pass class Foo(object): def __new__(cls,a): if a: return Bar(1,2,3) else: return super(Foo, cls).__new__(cls) Here is what happens in Python 2.3 CVS: >>> print Foo(0) <__main__.Foo object at 0x402baf0c> >>> print Foo(1) <__main__.Bar object at 0x402bab6c> All looks as it should, based on the discussions from when bug #537450 was last discussed. Here is what happens in Python 2.2.1 (and the current 2.2.2 CVS): >>> print Foo(0) <__main__.Foo object at 0x8174434> >>> print Foo(1) Traceback (most recent call last): File "", line 1, in ? TypeError: __init__() takes exactly 4 arguments (2 given) In the first case where a=0, the super-class new is called, and a Foo instance is returned. All is fine, so far. However, the second case returns an initialized Bar instance, which Foo tries to initialize again by calling Bar.__init__ with the same arguments that were passed into Foo.__new__. An arity error results, which is actually a good thing, since arbitrarily calling another object's constructor with arguments passed in for another object is a decidedly bad thing to do in general. The more severe problem occurs when the arguments do conform, and an instance of an improperly constructed object is returned. Either way, Bar is initialized _twice_, once with the intended arguments, another with arguments that were intended for Foo! I assert that no sensible developer would ever write code that relied on the pre-2.3 CVS semantics. I've done a survey of much of the meta-class code posted to or discussed on Python-dev and comp.lang.python, and none rely on this behavior. Plus, even if they did, we have already decided they will have to change it when 2.3 comes out. If you are not entirely convinced by the above argument, please consider a compromise solution: put the fix in for 2.2.1 Beta 1. If anyone has complaints based on extant code, then we can always back it out for Beta 2, or the final release. Thanks for taking the time... -Kevin Jacobs -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From mats@laplaza.org Sun Oct 6 18:45:44 2002 From: mats@laplaza.org (Mats Wichmann) Date: Sun, 06 Oct 2002 11:45:44 -0600 Subject: [Python-Dev] Re: [snake-farm] Autodetect /usr/ccs/bin? In-Reply-To: <20021006160005.8968.48937.Mailman@mail.python.org> Message-ID: <5.1.0.14.1.20021006114025.01dc2ca0@204.151.72.2> >[Anders Qvist] >> Several machines in the snake farm fail to build libpython2.3 because >> they don't have ar in their path. ar, ld and the like live in >> /usr/ccs/bin, which seems to be some kind of standard (for generous >> values of standard) as both SunOS 5.8, AIX 4.3 and HP/UX 11.00 seems >> ot have this dir. >> >> Should the configure script try with /usr/ccs/bin/ar and ld if none is >> explicitly supplied? > >This may be a good idea, but I don't know zit about this area. I'm >cc'ing python-dev, maybeo someone there understands this. > >--Guido van Rossum (home page: http://www.python.org/~guido/) /usr/ccs was a semi-standard location introduced in the System V days, probably not before SVR4. ccs = C Compilation System, don't know whether to blame Sun or USL for this innovation :-) which used to give me lots of support problems because nobody ever had these things in their path. These days it can be fun when people try to run gcc without binutils on Solaris, since again some key stuff may not be in the path. Don't see that it would hurt to have the configure script try there, perhaps limited by OS type? The only one of the above I know about is Solaris, though. From pedronis@bluewin.ch Sun Oct 6 20:02:05 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 6 Oct 2002 21:02:05 +0200 Subject: [Python-Dev] C3 implementation Message-ID: <031e01c26d6a$dcb4e9c0$6d94fea9@newmexico> I have an apparently working implementation of C3 in C. test_descr/test_descrtut pass apart some tests that involves hierarchies with MRO order disagreements. In such cases my impl raise TypeError: MRO order disagreement exceptions. Some of them it's clear to me how to fix them myself, for others someone should think about what was the original intention of the test. I can add some tests about monotonicity etc, with examples from the paper and the discussion we had here. Should I submit this as a patch on SF? at least is a start. PS: 1. the implementation is worst-case quadratic in the total length of the superclasses' mros, like the Python one I have presented here or the one used in Goo. I don't think this has relevance in practice, it's a trade-off versus even more complicated logic and data structure usage. From guido@python.org Sun Oct 6 20:40:48 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 06 Oct 2002 15:40:48 -0400 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: Your message of "Sun, 06 Oct 2002 13:13:30 EDT." References: Message-ID: <200210061940.g96Jem719413@pcp02138704pcs.reston01.va.comcast.net> You could have picked a better time than one day before the beta releast to open this up. I'm already asked to consider several patches (the Psyco ones) at the very last minute. In the SF entry I say that it's a clear semantic change and I don't want to change it in 2.2. I'll try to see if I can find time to read your message, but if I don't, I won't feel guilty. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sun Oct 6 20:44:12 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 06 Oct 2002 15:44:12 -0400 Subject: [Python-Dev] C3 implementation In-Reply-To: Your message of "Sun, 06 Oct 2002 21:02:05 +0200." <031e01c26d6a$dcb4e9c0$6d94fea9@newmexico> References: <031e01c26d6a$dcb4e9c0$6d94fea9@newmexico> Message-ID: <200210061944.g96JiC019534@pcp02138704pcs.reston01.va.comcast.net> > I have an apparently working implementation of C3 in C. Woo hoo! > test_descr/test_descrtut pass apart some tests that involves > hierarchies with MRO order disagreements. In such cases my impl > raise TypeError: MRO order disagreement exceptions. I'd like that to be a (new?) warning instead, so the user can control whether to turn it into an error or ignore it or fix it. > Some of them it's clear to me how to fix them myself, for others > someone should think about what was the original intention of the > test. > > I can add some tests about monotonicity etc, with examples from the > paper and the discussion we had here. > > Should I submit this as a patch on SF? at least is a start. Yes please! Assign to me, I'll get on it after I push 2.2.2 beta 1 out of the door. > PS: > 1. the implementation is worst-case quadratic in the total length of > the superclasses' mros, like the Python one I have presented here or > the one used in Goo. I don't think this has relevance in practice, > it's a trade-off versus even more complicated logic and data > structure usage. Sounds fine given the expected size of the average inheritance lattice. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Sun Oct 6 20:41:11 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 6 Oct 2002 21:41:11 +0200 Subject: [Python-Dev] Fw: MRO 2.2 vs 2.3 Message-ID: <038401c26d70$52fd6c60$6d94fea9@newmexico> ----- Original Message ----- From: "Robin Becker" Newsgroups: comp.lang.python Sent: Sunday, October 06, 2002 3:21 PM Subject: MRO 2.2 vs 2.3 > Will the C3 method resolution algorithm be back ported into Python > 2.2.x? It appears from the recently updated > > http://www.python.org/2.2.1/descrintro.html#mro > > that this will be used in 2.3, but the PBF version will then be out of > step with the future. > -- > Robin Becker From martin@v.loewis.de Sun Oct 6 20:53:38 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 06 Oct 2002 21:53:38 +0200 Subject: [Python-Dev] Re: [snake-farm] Autodetect /usr/ccs/bin? In-Reply-To: <200210061329.g96DTOs14718@pcp02138704pcs.reston01.va.comcast.net> References: <20021006115713.GB22400@lysator.liu.se> <200210061329.g96DTOs14718@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > > Several machines in the snake farm fail to build libpython2.3 because > > they don't have ar in their path. ar, ld and the like live in > > /usr/ccs/bin, which seems to be some kind of standard (for generous > > values of standard) as both SunOS 5.8, AIX 4.3 and HP/UX 11.00 seems > > ot have this dir. > > > > Should the configure script try with /usr/ccs/bin/ar and ld if none is > > explicitly supplied? > > This may be a good idea, but I don't know zit about this area. I'm > cc'ing python-dev, maybeo someone there understands this. Trying ar from /usr/ccs/bin might be reasonable, but it *must* try ar from PATH before that. I fail to see what common scenario this would fail for, though: If you don't have ar(1) in your path, how do you find make(1)? In no case configure should try /usr/ccs/bin/ld: Invoking ld directly is always incorrect (except for archaic systems). Regards, Martin From guido@python.org Sun Oct 6 21:30:21 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 06 Oct 2002 16:30:21 -0400 Subject: [Python-Dev] Anders Qvist: Re: [snake-farm] Autodetect /usr/ccs/bin? Message-ID: <200210062030.g96KULV28711@pcp02138704pcs.reston01.va.comcast.net> I've no time to research this further right now, so I'm forwarding Anders' mail here. It seems that Martin disagrees with Pike's solution though. --Guido van Rossum (home page: http://www.python.org/~guido/) ------- Forwarded Message Date: Sun, 06 Oct 2002 22:03:57 +0200 From: Anders Qvist To: Guido van Rossum Subject: Re: [snake-farm] Autodetect /usr/ccs/bin? On Sun, Oct 06, 2002 at 09:29:24AM -0400, Guido van Rossum wrote: > [Anders Qvist] > > Several machines in the snake farm fail to build libpython2.3 because > > they don't have ar in their path. ar, ld and the like live in > > /usr/ccs/bin, which seems to be some kind of standard (for generous > > values of standard) as both SunOS 5.8, AIX 4.3 and HP/UX 11.00 seems > > ot have this dir. > > > > Should the configure script try with /usr/ccs/bin/ar and ld if none is > > explicitly supplied? > > This may be a good idea, but I don't know zit about this area. I'm > cc'ing python-dev, maybeo someone there understands this. Apparently, the pike people do this. They seem to think it's the right thing to do. From their configure.in: case "$pike_cv_sys_os" in ... Solaris) LDSHARED="/usr/ccs/bin/ld -G";; ... They also add it to their PATH during various circumstances. Have a look at: http://community.roxen.com/_internal/cvsview!0/8223/1.633/0/configure.in - -- Anders "Quest" Qvist "We've all heard that a million monkeys banging on a million typewriters will eventually reproduce the entire works of Shakespeare. Now, thanks to the Internet, we know this is not true." -- Robert Wilensky ------- End of Forwarded Message From jacobs@penguin.theopalgroup.com Sun Oct 6 21:31:40 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Sun, 6 Oct 2002 16:31:40 -0400 (EDT) Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: <200210061940.g96Jem719413@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Sun, 6 Oct 2002, Guido van Rossum wrote: > You could have picked a better time than one day before the beta > releast to open this up. I'm already asked to consider several > patches (the Psyco ones) at the very last minute. Sorry -- it was on the 2.2.2 wiki since the day it was created. Plus, I've been out of town much of the time since then. > In the SF entry I say that it's a clear semantic change and I don't > want to change it in 2.2. > I'll try to see if I can find time to read your message, but if I > don't, I won't feel guilty. I would appreciate it. This is something of a pet bug, since it is the single thing in 2.2.2 that will break a fairly large application that we have developed. For somewhat complex reasons, there is no easy work-around for it. Thanks, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From guido@python.org Sun Oct 6 21:57:36 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 06 Oct 2002 16:57:36 -0400 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: Your message of "Sun, 06 Oct 2002 16:31:40 EDT." References: Message-ID: <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> > > You could have picked a better time than one day before the beta > > releast to open this up. I'm already asked to consider several > > patches (the Psyco ones) at the very last minute. > > Sorry -- it was on the 2.2.2 wiki since the day it was created. > Plus, I've been out of town much of the time since then. Sigh. Wikis have one big problem: unless you check them regularly it's easy to miss a discussion going on there. I'm sorry about that -- I've completely failed to check the Wiki. BTW, the Wiki is here: http://manatee.mojam.com/py222wiki/ I tried to set up an email subscription but the Wiki isn't configured to support that... > > In the SF entry I say that it's a clear semantic change and I don't > > want to change it in 2.2. > > > I'll try to see if I can find time to read your message, but if I > > don't, I won't feel guilty. > > I would appreciate it. This is something of a pet bug, since it is > the single thing in 2.2.2 that will break a fairly large application > that we have developed. For somewhat complex reasons, there is no > easy work-around for it. I've now read your whole message. I reluctantly agree that it's reasonable to fix this and will try to get this in the beta. These are the outstanding issues for the beta that I'm aware of now: - 537450 (this) - three patches from Armin Rigo to make life easier for Psyco - test_email fails on Windows - patch 619352: Updated .spec file for 2.2.2b1 - whatever's in the Wiki --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs@penguin.theopalgroup.com Sun Oct 6 22:05:41 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Sun, 6 Oct 2002 17:05:41 -0400 (EDT) Subject: [Python-Dev] Anders Qvist: Re: [snake-farm] Autodetect /usr/ccs/bin? In-Reply-To: <200210062030.g96KULV28711@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Sun, 6 Oct 2002, Guido van Rossum wrote: > ------- Forwarded Message > > Date: Sun, 06 Oct 2002 22:03:57 +0200 > From: Anders Qvist > To: Guido van Rossum > Subject: Re: [snake-farm] Autodetect /usr/ccs/bin? > > Apparently, the pike people do this. They seem to think it's the right > thing to do. From their configure.in: > > case "$pike_cv_sys_os" in > ... > Solaris) LDSHARED="/usr/ccs/bin/ld -G";; > ... > > They also add it to their PATH during various circumstances. Have a > look at: I _know_ this is wrong for several non-native (non-SunPro) Solaris compilers. My applications assume that $PATH has the correct ld/ar/cpp. If not, we augment the search to look in /usr/ccs, /opt/sunWPro, and other more obscure locations. I believe this is in accord with Martin suggestion. Regards, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From Raymond Hettinger" <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <003001c26d7e$3ce5df80$890ca044@oemcomputer> From: "Guido van Rossum" > These are the outstanding issues for the beta that I'm aware of now: > > - 537450 (this) > - three patches from Armin Rigo to make life easier for Psyco > - test_email fails on Windows > - patch 619352: Updated .spec file for 2.2.2b1 > - whatever's in the Wiki The last one includes a list of modules that still need backport attention (by someone more knowledgable than me): - Doc/ref/ref5.tex - Doc/ref/ref7.tex - Lib/socket.py - Lib/pydoc.py - Lib/pickle.py - Lib/posixpath.py Also, I've made a large number of documentation backports since Fred rebuilt the docs on Friday. So, one more doc build is in order before the beta goes out. Raymond Hettinger From martin@v.loewis.de Sun Oct 6 22:23:28 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 06 Oct 2002 23:23:28 +0200 Subject: [Python-Dev] Anders Qvist: Re: [snake-farm] Autodetect /usr/ccs/bin? In-Reply-To: <200210062030.g96KULV28711@pcp02138704pcs.reston01.va.comcast.net> References: <200210062030.g96KULV28711@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > Apparently, the pike people do this. They seem to think it's the right > thing to do. From their configure.in: > > case "$pike_cv_sys_os" in > ... > Solaris) LDSHARED="/usr/ccs/bin/ld -G";; > ... > > They also add it to their PATH during various circumstances. Have a > look at: Using ld directly is wrong because it fails to add implicit linker options, in particular crt*.o. For example, on SCO, giving -Kpthread to the compiler implies giving -lpthread -l to the linker. Likewise, using gcc on Solaris, linking with gcc implies linking crtbegin.o and crtend.o, which is necessary to run constructors of global C++ objects. The same holds for SunPR, which implicitly links crti.o, crt1.o, values-xa.o, and crtn.o for an executable, and crti.o, values-xa.o, and crtn.o for a shared object. Regards, Martin From martin@v.loewis.de Sun Oct 6 22:30:52 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 06 Oct 2002 23:30:52 +0200 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> References: <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > - whatever's in the Wiki I'd suggest to ignore the Wiki. If the PBF ever manages to act, there will be a 2.2.3 release. Only consider things that just require a decision; if you find that things require work beyond applying a right-looking patch, have proponents do the work for 2.2.3. Regards, Martin From martin@v.loewis.de Sun Oct 6 22:34:04 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 06 Oct 2002 23:34:04 +0200 Subject: [Python-Dev] Anders Qvist: Re: [snake-farm] Autodetect /usr/ccs/bin? In-Reply-To: References: Message-ID: Kevin Jacobs writes: > I _know_ this is wrong for several non-native (non-SunPro) Solaris > compilers. My applications assume that $PATH has the correct ld/ar/cpp. If > not, we augment the search to look in /usr/ccs, /opt/sunWPro, and other more > obscure locations. I believe this is in accord with Martin suggestion. Indeed it is - although I'd be surprised to find ar in /opt/SUNWpro (that locations has no ar in my installation). The usual comment for ld applies (don't ever use it); for cpp, cc -E should work just as well. Regards, Martin From martin@v.loewis.de Sun Oct 6 22:36:12 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 06 Oct 2002 23:36:12 +0200 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: <003001c26d7e$3ce5df80$890ca044@oemcomputer> References: <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> <003001c26d7e$3ce5df80$890ca044@oemcomputer> Message-ID: "Raymond Hettinger" writes: > The last one includes a list of modules that still need > backport attention (by someone more knowledgable than me): Do those need to happen for 2.2.2, or would it be acceptable if whatever problems they fix get only fixed in 2.2.3? I think it is of high importance that 2.2.2 is released soon; anything that delays 2.2.2 must be of really high importance. Regards, Martin From Raymond Hettinger" <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net><003001c26d7e$3ce5df80$890ca044@oemcomputer> Message-ID: <007401c26d81$59f914e0$890ca044@oemcomputer> > "Raymond Hettinger" writes: > > > The last one includes a list of modules that still need > > backport attention (by someone more knowledgable than me): > > Do those need to happen for 2.2.2, or would it be acceptable if > whatever problems they fix get only fixed in 2.2.3? > > I think it is of high importance that 2.2.2 is released soon; anything > that delays 2.2.2 must be of really high importance. Yes, this is a list for 2.2.2 based on my review of the checkin logs. If I felt comfortable applying these myself, they would already be done. There are about 8 or 9 total changes, each of them are small (a few lines in a single file), and can be done in an hour by someone who knows what they're doing. If only one or two gets looked at, pickle.py and socket.py have the most important unapplied patches. Raymond Hettinger From guido@python.org Sun Oct 6 23:13:14 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 06 Oct 2002 18:13:14 -0400 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: Your message of "Sun, 06 Oct 2002 17:43:03 EDT." <007401c26d81$59f914e0$890ca044@oemcomputer> References: <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> <003001c26d7e$3ce5df80$890ca044@oemcomputer> <007401c26d81$59f914e0$890ca044@oemcomputer> Message-ID: <200210062213.g96MDER30213@pcp02138704pcs.reston01.va.comcast.net> > Yes, this is a list for 2.2.2 based on my review of the checkin logs. > If I felt comfortable applying these myself, they would already > be done. There are about 8 or 9 total changes, each of them > are small (a few lines in a single file), and can be done in an > hour by someone who knows what they're doing. > > If only one or two gets looked at, pickle.py and socket.py > have the most important unapplied patches. Can you mail a list of which checkins specifically you are looking at for those files that you want to be reviewed? --Guido van Rossum (home page: http://www.python.org/~guido/) From greg@cosc.canterbury.ac.nz Sun Oct 6 23:15:07 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Mon, 07 Oct 2002 11:15:07 +1300 (NZDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <15773.1109.273430.632460@localhost.localdomain> Message-ID: <200210062215.g96MF7F19321@oma.cosc.canterbury.ac.nz> Greg> Maybe if ":" were a sliceobject-creating operator usable anwyere, Greg> not just in [...], and you added arithmetic methods to Greg> sliceobjects so you could use them as rationals... Skip> Does x[1:4] return a slice out of x or x[rational(1,4)]? If rationals and slice objects were the same thing, it wouldn't matter! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido@python.org Sun Oct 6 23:18:55 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 06 Oct 2002 18:18:55 -0400 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 Message-ID: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> The file pep_2777.py uses an encoding cookie that specifies UTF-8. Unfortunately my toolchain doesn't know about this, and displays it as Latin-1. Since the only UTF-8 is in 8-bit string literals (not Unicode literals), wouldn't it make more sense to drop the encoding cookie and use \xXX escapes in those literals? I'm not even sure this use is legal in phase 2 of PEP 263. --Guido van Rossum (home page: http://www.python.org/~guido/) From Raymond Hettinger" <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> <003001c26d7e$3ce5df80$890ca044@oemcomputer> <007401c26d81$59f914e0$890ca044@oemcomputer> <200210062213.g96MDER30213@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <00e701c26d87$d0f317c0$890ca044@oemcomputer> [GvR] > Can you mail a list of which checkins specifically you are looking at > for those files that you want to be reviewed? == Socket.py == Revision 1.35 :On Windows, make sure SocketType is the same as socket. (SF bug 598097) Revision 1.34: Delete junk attributes left behind by _socketobject class construction. Revision 1.33: The _socketobject class has no need for a __del__ method: all it did was to delete the reference to self._sock, and the regular destructor will do that just fine. This made some hacks in close() unnecessary. The _fileobject class still has a __del__ method, because it must flush. Revision 1.32: OK, one more hack: speed up the case of readline() in unbuffered mode. This is important IMO because httplib reads the headers this way. == Pickle.py == Revision 1.70: whichmodule() should skip dummy package entries in sys.modules. This fixes the charming, but unhelpful error message for >>> pickle.dumps(type.__new__) Can't pickle : it's not the same object as datetime.math.__new__ Bugfix candidate. Revision 1.69: Patch #505705: Remove eval in pickle and cPickle. == pydoc.py == Revision 1.63 :In both spilldata() functions, pretend that the docstring for non-callable objects is always None. This makes for less confusing output and fixes the problem reported in SF patch #550290. Revision 1.68: Extend stripid() to handle strings ending in more than one '>'. Add resolve() to handle looking up objects and names (fix SF bug 586931). Add a nicer error message when given a filename that doesn't exist. == ref5.tex == Revision 1.63: Simplify the production for argument list, making sure that it actually allows all the legal syntax, and nothing else. Previously, it did not allow a call like func(arg, **dictionary). This closes (again!) SF bug #493243. Revision 1.62: Played contortionist games with the argument_list production so it might be easier to understand. This relates to SF bug #493243, which will be closed. From mhammond@skippinet.com.au Mon Oct 7 00:39:17 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Mon, 7 Oct 2002 09:39:17 +1000 Subject: [Python-Dev] Unclear on the way forward with unsigned integers Message-ID: I'm a little confused by the new world order for working with integers in extension modules. At the end of the day, my question is this: Assume my extension module has an unsigned integer it wishes to return to Python. Further, assume that this unsigned integer is not really an integer as such, but more a set of bits, or some other value "encoded" in 32 bits, such as an enum. (To put it another way, the "signedness" of this value seems more random than chosen) How should I create the object to return to Python? For a concrete example. Let's say I want to return the value of the Win32 function GetVersion(). The documentation for this function declares it is an unsigned 32 bit value. The documentation then explains that to decode this value, specific bits in the value should be examined. It then expounds on this with C sample code that relies on this unsigned behaviour by using a simple "> 0x80000000" comparison to check the high bit! I see 2 choices for returning this value: * Use PyInt_FromLong() - this will give me a *signed* Python integer, but with an identical bit pattern. * Use PyLong_FromUnsignedLong() - this function will correctly be signed, but may no longer fit in 32 bits. Now, I think I am trying to stay too close to the hardware for a language like Python, but something just seems wrong with promoting my nice 32 bit value to a Python long, simply for the sake of retaining the sign for a value that the whole concept of "signed" doesn't make much sense (as it doesn't in this case, or in the case of enums etc). Any suggestions or general advice? While this case seems quite trivial, I am starting to face this issue more and more, especially as I am seeing these lovely "FutureWarnings" from all my lovely 32 bit hexadecimal constants Thanks, Mark. From thomas@xs4all.net Mon Oct 7 02:01:02 2002 From: thomas@xs4all.net (Thomas Wouters) Date: Mon, 7 Oct 2002 03:01:02 +0200 Subject: [Python-Dev] Unclear on the way forward with unsigned integers In-Reply-To: References: Message-ID: <20021007010102.GC5116@xs4all.nl> On Mon, Oct 07, 2002 at 09:39:17AM +1000, Mark Hammond wrote: > Assume my extension module has an unsigned integer it wishes to return to > Python. Further, assume that this unsigned integer is not really an integer > as such, but more a set of bits, or some other value "encoded" in 32 bits, > such as an enum. (To put it another way, the "signedness" of this value > seems more random than chosen) > I see 2 choices for returning this value: > * Use PyInt_FromLong() - this will give me a *signed* Python integer, but > with an identical bit pattern. If you intend to hand out the bitvector for informational purposes, and aren't too worried (because of your excellent documentational powers) about users messing up the bitvector in incorrect calculations before passing it back, or about them being dazzled by the concept of bitvectors as a whole, go for it. I'm personally still in that phase where I really like ANDing and ORing, even in Python :) > * Use PyLong_FromUnsignedLong() - this function will correctly be signed, > but may no longer fit in 32 bits. You mean on the way back, I presume ? > Now, I think I am trying to stay too close to the hardware for a language > like Python, Yes, I think you are, actually. Converting those cute little 32 bits into a real Python object representing it more verbosely, so to speak, might seem a waste, but makes it a lot more user-friendly. It doesn't have to be a type on its own, it can be a tuple of strings or ints, depending on what kind of info you want to return. If it's really a plain bitvector, you can also consider returning a string. A hobby project of mine, a type of MUD, converts internal bitvectors to strings where each letter from 'a' to 'F' represents a bit in the bitvector. So 0x8a would come out as "bdh". This is very suited to the project itself, since the internal language (MUSHcode, an ass-backwards retarded vaguely-lisplike death-trap of a recursively-parsed scripting language) has no bitwise operators, and the typical MUSHcoder has trouble grasping bitwise arithmatic anyway. Not that MUSHcode has an efficient way to say '"a" in "abc"', mind you, but they manage to do it in less than 5 functioncalls. (And yes, I am trying to replace MUSHcode with Python.) I should point out that neither the bitvector-to-string code (and its reverse counterpart) nor the braindead internal language are mine :) I just inherited the project. But the bitvector routines are simple, I'm sure you'll have no trouble figuring them out ;) When using them in Python code you might want to add the flag-characters as constants in your module, so the test becomes: if ce.hasdisk in ce.systemflags(): Insomaniac'ly y'rs, -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! From ark@research.att.com Mon Oct 7 02:32:52 2002 From: ark@research.att.com (Andrew Koenig) Date: 06 Oct 2002 21:32:52 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210062215.g96MF7F19321@oma.cosc.canterbury.ac.nz> References: <200210062215.g96MF7F19321@oma.cosc.canterbury.ac.nz> Message-ID: Skip> Does x[1:4] return a slice out of x or x[rational(1,4)]? Greg> If rationals and slice objects were the same thing, Greg> it wouldn't matter! But then how would you distinguish between x[1:4] and x[2:8]? -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From greg@cosc.canterbury.ac.nz Mon Oct 7 02:39:25 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Mon, 07 Oct 2002 14:39:25 +1300 (NZDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Message-ID: <200210070139.g971dPs19874@oma.cosc.canterbury.ac.nz> Greg> If rationals and slice objects were the same thing, Greg> it wouldn't matter! Andrew> But then how would you distinguish between x[1:4] and x[2:8]? Well, if you reduced the rational to lowest terms you would have to keep the original values around for when you were using it as a slice object... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one@comcast.net Mon Oct 7 02:43:21 2002 From: tim.one@comcast.net (Tim Peters) Date: Sun, 06 Oct 2002 21:43:21 -0400 Subject: [Python-Dev] Unclear on the way forward with unsigned integers In-Reply-To: Message-ID: [Mark Hammond] > I'm a little confused by the new world order for working with integers > in extension modules. > > At the end of the day, my question is this: > > Assume my extension module has an unsigned integer it wishes to return > to Python. Further, assume that this unsigned integer is not really > an integer as such, but more a set of bits, or some other value > "encoded" in 32 bits, such as an enum. (To put it another way, the > "signedness" of this value seems more random than chosen) Well, it matters: a bitset is most naturally thought of as unsigned, so that shifting doesn't introduce "mystery bits". OTOH, a C enum is, by definition, a signed integer. > How should I create the object to return to Python? I'd create a Python long. > For a concrete example. Let's say I want to return the value of the > Win32 function GetVersion(). > > The documentation for this function declares it is an unsigned 32 > bit value. The documentation then explains that to decode this value, > specific bits in the value should be examined. It then expounds on > this with C sample code that relies on this unsigned behaviour by > using a simple "> 0x80000000" comparison to check the high bit! Yup. The docs also say : This function has been superseded by GetVersionEx, which is the preferred method for obtaining system version number information. New applications should use GetVersionEx. The GetVersionEx function was developed because many existing applications err when examining the DWORD return value of a GetVersion function call, transposing the major and minor version numbers packed into that DWORD. IOW, the bag-of-bits mixed-with bag-of-bytes model was too confusing to work with. > I see 2 choices for returning this value: > > * Use PyInt_FromLong() - this will give me a *signed* Python integer, > but with an identical bit pattern. > > * Use PyLong_FromUnsignedLong() - this function will correctly be > signed, but may no longer fit in 32 bits. Python ints don't fit in 32 bits either: they've got object headers like all objects have. The space difference here is trivial. A third choice is to pick the values apart in C, delivering a (NT_or_later_bool, major_version_int, minor_version_int, build_int) tuple back to the Python user. Making people pick apart the bits in Python code seems too low-level here: dwVersion = GetVersion(); // Get major and minor version numbers of Windows dwWindowsMajorVersion = (DWORD)(LOBYTE(LOWORD(dwVersion))); dwWindowsMinorVersion = (DWORD)(HIBYTE(LOWORD(dwVersion))); // Get build numbers for Windows NT or Win32s if (dwVersion < 0x80000000) // Windows NT dwBuild = (DWORD)(HIWORD(dwVersion)); else if (dwWindowsMajorVersion < 4) // Win32s dwBuild = (DWORD)(HIWORD(dwVersion) & ~0x8000); else // Windows 95 -- No build numbers provided dwBuild = 0; > Now, I think I am trying to stay too close to the hardware for a > language like Python, but something just seems wrong with promoting > my nice 32 bit value to a Python long, simply for the sake of > retaining the sign for a value that the whole concept of "signed" > doesn't make much sense (as it doesn't in this case, or in the case > of enums etc). Except Python doesn't *have* unsigned ints, so the only faithful way to return one is to make a Python long. In this specific case, though, I think it would be better to pick the bits apart *for* the user -- there's really no use for the raw int, signed or unsigned, except after picking it apart. > Any suggestions or general advice? While this case seems quite > trivial, I am starting to face this issue more and more, especially > as I am seeing these lovely "FutureWarnings" from all my lovely 32 > bit hexadecimal constants Sticking "L" at the end is usually all it takes. From martin@v.loewis.de Mon Oct 7 03:32:42 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 07 Oct 2002 04:32:42 +0200 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: <00e701c26d87$d0f317c0$890ca044@oemcomputer> References: <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> <003001c26d7e$3ce5df80$890ca044@oemcomputer> <007401c26d81$59f914e0$890ca044@oemcomputer> <200210062213.g96MDER30213@pcp02138704pcs.reston01.va.comcast.net> <00e701c26d87$d0f317c0$890ca044@oemcomputer> Message-ID: "Raymond Hettinger" writes: > Revision 1.69: Patch #505705: Remove eval in pickle and cPickle. It is not realistic to backport this patch for 2.2.2. I would suggest to ignore it completely for 2.2. Regards, Martin From Raymond Hettinger" <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net><003001c26d7e$3ce5df80$890ca044@oemcomputer><007401c26d81$59f914e0$890ca044@oemcomputer><200210062213.g96MDER30213@pcp02138704pcs.reston01.va.comcast.net><00e701c26d87$d0f317c0$890ca044@oemcomputer> Message-ID: <00ab01c26daa$e30f64e0$890ca044@oemcomputer> Martin, thanks for looking at it. BTW, I'm not a proponent of any of those changes. They are just ones that looked like viable candidates but I didn't think I should handle myself. Raymond Hettinger ----- Original Message ----- From: "Martin v. Loewis" To: "Raymond Hettinger" Cc: "Guido van Rossum" ; Sent: Sunday, October 06, 2002 10:32 PM Subject: Re: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ > "Raymond Hettinger" writes: > > > Revision 1.69: Patch #505705: Remove eval in pickle and cPickle. > > It is not realistic to backport this patch for 2.2.2. I would suggest > to ignore it completely for 2.2. > > Regards, > Martin From martin@v.loewis.de Mon Oct 7 03:41:09 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 07 Oct 2002 04:41:09 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > The file pep_2777.py uses an encoding cookie that specifies UTF-8. > Unfortunately my toolchain doesn't know about this, and displays it as > Latin-1. What do you mean by "toolchain"? At the end of the chain is python, it should know about this well enough. Did you try to open the file in IDLE? What other tools are you using? > Since the only UTF-8 is in 8-bit string literals (not Unicode > literals), wouldn't it make more sense to drop the encoding cookie > and use \xXX escapes in those literals? Having the original byte strings allows to verify correctness of the test visually: the files created should look the same in the operating system (using ls, or a file explorer) as they do in the source code. > I'm not even sure this use is legal in phase 2 of PEP 263. It is: the file has a decoding declared, and decodes properly under that decoding. String literals are preserved in their original encoding, so they will come out at run-time the same as they are on disk. Regards, Martin From jafo@tummy.com Mon Oct 7 05:17:06 2002 From: jafo@tummy.com (Sean Reifschneider) Date: Sun, 6 Oct 2002 22:17:06 -0600 Subject: [Python-Dev] Patch to fix Distutils bdist_rpm on Red Hat 8.0 Message-ID: <20021007041706.GA11457@tummy.com> At the 8.0 release, Red Hat has split the "rpm" command such that the build options reside only in the "rpmbuild" command. Actually, this happened a while ago, as "rpmbuild" was available on 7.3. However, at 8.0 the "rpm" command would no longer do building and it fails out. I've submitted patch #619493 to correct this. If /usr/bin/rpmbuild or /bin/rpmbuild exists, the rpm command is changed from "rpm" to "rpmbuild". This probably should go into the 2.2.2 release. I'm submitting a bug to Red Hat as well, as they probably should release a fixed python package to address this as well. Sean -- "The phrase ``ship it!'' is one with long and deep resonances in my benighted and antiquated big-software-dept career." -- John Shipman, 1998 Sean Reifschneider, Inimitably Superfluous tummy.com, ltd. - Linux Consulting since 1995. Qmail, Python, SysAdmin From tim@multitalents.net Mon Oct 7 05:54:44 2002 From: tim@multitalents.net (Tim Rice) Date: Sun, 6 Oct 2002 21:54:44 -0700 (PDT) Subject: [Python-Dev] Re: [snake-farm] Autodetect /usr/ccs/bin? In-Reply-To: <200210061329.g96DTOs14718@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Sun, 6 Oct 2002, Guido van Rossum wrote: > [Anders Qvist] > > Several machines in the snake farm fail to build libpython2.3 because > > they don't have ar in their path. ar, ld and the like live in > > /usr/ccs/bin, which seems to be some kind of standard (for generous > > values of standard) as both SunOS 5.8, AIX 4.3 and HP/UX 11.00 seems > > ot have this dir. > > > > Should the configure script try with /usr/ccs/bin/ar and ld if none is > > explicitly supplied? > How about? AC_PATH_PROG(AR, ar,[AC_MSG_ERROR(Can't find ar in your path)]) > This may be a good idea, but I don't know zit about this area. I'm > cc'ing python-dev, maybeo someone there understands this. > > --Guido van Rossum (home page: http://www.python.org/~guido/) > -- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net From mal@lemburg.com Mon Oct 7 08:56:12 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 07 Oct 2002 09:56:12 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3DA13E1C.5080505@lemburg.com> Guido van Rossum wrote: > The file pep_2777.py uses an encoding cookie that specifies UTF-8. > Unfortunately my toolchain doesn't know about this, and displays it as > Latin-1. Since the only UTF-8 is in 8-bit string literals (not > Unicode literals), wouldn't it make more sense to drop the encoding > cookie and use \xXX escapes in those literals? You could, but the whole point of PEP 263 is that the source code encoding is made explicit, so removing the cookie would defeat the purpose. Note that the cookie also implicitly tests the PEP 263 implementation. > I'm not even sure this > use is legal in phase 2 of PEP 263. It is: if you use non-UTF-8 string literals in that source file, the parser will raise an exception during compilation. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From vinay_sajip@red-dove.com Mon Oct 7 09:16:29 2002 From: vinay_sajip@red-dove.com (Vinay Sajip) Date: Mon, 7 Oct 2002 09:16:29 +0100 Subject: [Python-Dev] PEP 282 Implementation References: <00e001c2261d$19bfc320$652b6992@alpha> <200208092040.g79Ke3S31416@pcp02138704pcs.reston01.va.comcast.net> <006601c24e7f$fcff5440$652b6992@alpha> <200209231741.g8NHfEl11092@pcp02138704pcs.reston01.va.comcast.net> <01aa01c26379$8d6def60$4901000a@dorothy> <003901c2639f$e7e21ae0$652b6992@alpha> <200209241537.g8OFbnc13228@odiug.zope.com> Message-ID: <002201c26dd9$d7c46b60$652b6992@alpha> Guido van Rossum wrote: >> Chris McDonough wrote: >>> It would be helpful for the FileHandler class to define a method >>> which just closes and reopens the current logfile (instead of >>> actually rotating a set like-named logfiles). This would allow >>> logfile rotation to be performed by a separate process (e.g. >>> RedHat's logrotate). Sometimes it's better (and even necessary) to >>> be able to use system-provided log rotation facilities instead of >>> relying on the native rotation facilities. > Having looked at it again, I think that this is definitely better than > doing log rotation in the FileHandler. The rotation code in the log > handler currently calls tell() after each record is emitted. This is > expensive, and not needed if you use an external process to watch over > the log files and rotate them. I think file rotation can definitely come out of the core, since there are many ways to do it and no one "right" way. I'll move the rotation stuff into a subclass of FileHandler called RotatingFileHandler (anyone think of a better name?). The FileHandler will remain in the core module; the subclass will be in the "logging.handlers" module. Vinay From mwh@python.net Mon Oct 7 10:30:49 2002 From: mwh@python.net (Michael Hudson) Date: 07 Oct 2002 10:30:49 +0100 Subject: [Python-Dev] Python 2.2.2 beta release on Monday! In-Reply-To: Guido van Rossum's message of "Sat, 05 Oct 2002 03:18:47 -0400" References: <200210050718.g957IlF01399@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <2m65wesiie.fsf@starship.python.net> Guido van Rossum writes: > I plan to do a Python 2.2.2 beta release this Monday (if Fred's and > Tim's time is available to do their part). > > Please finish your changes to 2.2.2 before the weekend is over! Timezones to the rescue... there are definitely a few things I want to do before a beta release. there-goes-the-coffee-break-ly y'rs M. -- In case you're not a computer person, I should probably point out that "Real Soon Now" is a technical term meaning "sometime before the heat-death of the universe, maybe". -- Scott Fahlman From thomas.heller@ion-tof.com Mon Oct 7 11:07:11 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon, 7 Oct 2002 12:07:11 +0200 Subject: [Python-Dev] Unclear on the way forward with unsigned integers References: Message-ID: <00ca01c26de9$4dbcedb0$e000a8c0@thomasnotebook> > Except Python doesn't *have* unsigned ints, so the only faithful way to > return one is to make a Python long. In this specific case, though, I think > it would be better to pick the bits apart *for* the user -- there's really > no use for the raw int, signed or unsigned, except after picking it apart. > > > Any suggestions or general advice? While this case seems quite > > trivial, I am starting to face this issue more and more, especially > > as I am seeing these lovely "FutureWarnings" from all my lovely 32 > > bit hexadecimal constants > [Tim] > Sticking "L" at the end is usually all it takes. That removes the warnings for 'x = 0x80000000L'. Is there a way (other than the -w command-line arg) to suppress the warnings when doing 'hex(-1)'? Shouldn't there be a __future__ option? Thomas From guido@python.org Mon Oct 7 13:20:27 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 08:20:27 -0400 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: Your message of "Mon, 07 Oct 2002 04:41:09 +0200." References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210071220.g97CKRG06496@pcp02138704pcs.reston01.va.comcast.net> > > The file pep_2777.py uses an encoding cookie that specifies UTF-8. > > Unfortunately my toolchain doesn't know about this, and displays it as > > Latin-1. > > What do you mean by "toolchain"? At the end of the chain is python, it > should know about this well enough. > > Did you try to open the file in IDLE? What other tools are you using? XEmacs. The particular version or configuration I'm using apparently doesn't recogize the coding cookie. > > Since the only UTF-8 is in 8-bit string literals (not Unicode > > literals), wouldn't it make more sense to drop the encoding cookie > > and use \xXX escapes in those literals? > > Having the original byte strings allows to verify correctness of the > test visually: the files created should look the same in the operating > system (using ls, or a file explorer) as they do in the source code. You could do this with a comment. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 7 13:28:24 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 08:28:24 -0400 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: Your message of "Mon, 07 Oct 2002 09:56:12 +0200." <3DA13E1C.5080505@lemburg.com> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> Message-ID: <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> > You could, but the whole point of PEP 263 is that the source code > encoding is made explicit, so removing the cookie would defeat the > purpose. Note that the cookie also implicitly tests the PEP 263 > implementation. I'm still not 100% comfortable with using arbitrary coding cookies in the Python distribution. I like having the feature, but I would prefer if the Python source distribution could be viewed with tools that aren't encoding-aware. On my Red Hat 7.2 Linux box, I have found no programs that understand the UTF8 cookie. They all seem to default to Latin-1. Since I also don't really want to assume Latin-1, I'd prefer if we could stick to ASCII, with the exception of tests explicitly needed for PEP 263. I don't want to make test_pep277.py an implicit PEP 263 test. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Mon Oct 7 14:02:10 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 07 Oct 2002 15:02:10 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3DA185D2.3010100@lemburg.com> Guido van Rossum wrote: >>You could, but the whole point of PEP 263 is that the source code >>encoding is made explicit, so removing the cookie would defeat the >>purpose. Note that the cookie also implicitly tests the PEP 263 >>implementation. > > > I'm still not 100% comfortable with using arbitrary coding cookies in > the Python distribution. I like having the feature, but I would > prefer if the Python source distribution could be viewed with tools > that aren't encoding-aware. On my Red Hat 7.2 Linux box, I have found > no programs that understand the UTF8 cookie. I just checked my SuSE 8.0 XEmacs and it seems that it doesn't come with the UTF-8 coding system installed either. > They all seem to default > to Latin-1. Since I also don't really want to assume Latin-1, I'd > prefer if we could stick to ASCII, with the exception of tests > explicitly needed for PEP 263. I don't want to make test_pep277.py an > implicit PEP 263 test. Fine with me. I guess we ought to document this somewhere in the Python core programming style guide (if such a document exists ;-). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Mon Oct 7 14:23:12 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 09:23:12 -0400 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: Your message of "Sun, 06 Oct 2002 18:29:20 EDT." <00e701c26d87$d0f317c0$890ca044@oemcomputer> References: <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> <003001c26d7e$3ce5df80$890ca044@oemcomputer> <007401c26d81$59f914e0$890ca044@oemcomputer> <200210062213.g96MDER30213@pcp02138704pcs.reston01.va.comcast.net> <00e701c26d87$d0f317c0$890ca044@oemcomputer> Message-ID: <200210071323.g97DNCH08002@pcp02138704pcs.reston01.va.comcast.net> > == Socket.py == > > Revision 1.35 :On Windows, make sure SocketType is the same as socket. (SF bug > 598097) > > Revision 1.34: Delete junk attributes left behind by _socketobject class construction. > > Revision 1.33: The _socketobject class has no need for a __del__ method: all it did was > to delete the reference to self._sock, and the regular destructor will > do that just fine. This made some hacks in close() unnecessary. > The _fileobject class still has a __del__ method, because it must flush. > > Revision 1.32: OK, one more hack: speed up the case of readline() in unbuffered mode. > This is important IMO because httplib reads the headers this way. Most of these build upon major restructuring of socket.py that I did for 2.3. Backporting would be too much of a hassle, so I'll skip them all. > == Pickle.py == > > Revision 1.70: whichmodule() should skip dummy package entries in sys.modules. > This fixes the charming, but unhelpful error message for > >>> pickle.dumps(type.__new__) > Can't pickle : it's not the same object as datetime.math.__new__ > Bugfix candidate. Fixed (together with 1.71 which was a cleanup of 1.70). > Revision 1.69: Patch #505705: Remove eval in pickle and cPickle. MvL already said this is a no-go; it depends on a new encoding that knows about string literal escape sequences. > == pydoc.py == > > Revision 1.63 :In both spilldata() functions, pretend that the docstring for > non-callable objects is always None. This makes for less confusing > output and fixes the problem reported in SF patch #550290. This applies cleanly so I'll check it in. > Revision 1.68: Extend stripid() to handle strings ending in more than one '>'. > Add resolve() to handle looking up objects and names (fix SF bug 586931). > Add a nicer error message when given a filename that doesn't exist. Looks messy, I'm not an expert on pydoc. Ping, should this be backported? > == ref5.tex == > > Revision 1.63: Simplify the production for argument list, making sure that it > actually allows all the legal syntax, and nothing else. Previously, > it did not allow a call like func(arg, **dictionary). > This closes (again!) SF bug #493243. > > Revision 1.62: Played contortionist games with the argument_list production so it > might be easier to understand. > This relates to SF bug #493243, which will be closed. I've forwarded these two to Fred Drake. --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz@pythoncraft.com Mon Oct 7 14:25:42 2002 From: aahz@pythoncraft.com (Aahz) Date: Mon, 7 Oct 2002 09:25:42 -0400 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: <3DA185D2.3010100@lemburg.com> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> Message-ID: <20021007132542.GA25844@panix.com> On Mon, Oct 07, 2002, M.-A. Lemburg wrote: > Guido van Rossum wrote: >> >>They all seem to default >>to Latin-1. Since I also don't really want to assume Latin-1, I'd >>prefer if we could stick to ASCII, with the exception of tests >>explicitly needed for PEP 263. I don't want to make test_pep277.py an >>implicit PEP 263 test. > > Fine with me. I guess we ought to document this somewhere in the > Python core programming style guide (if such a document exists ;-). PEP 8 -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Project Vote Smart: http://www.vote-smart.org/ From list-python@ccraig.org Mon Oct 7 14:28:28 2002 From: list-python@ccraig.org (Christopher A. Craig) Date: 07 Oct 2002 09:28:28 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <000001c26d53$64253b70$6701a8c0@NICKLEBY> References: <000001c26d53$64253b70$6701a8c0@NICKLEBY> Message-ID: "Paul F Dubois" writes: > I haven't followed the rational discussion but obviously there is no > other long-standing and highly used notation for rationals except > 1.0/2.0, is there? Kirby Urner suggested 1r2 as a way of expressing rationals, which I rather liked, but no I don't know of a traditional notation for rationals. -- Christopher A. Craig "Bah! Puny, weak newsreaders!"- GNUS info, in reference to 'global' kills From guido@python.org Mon Oct 7 14:41:02 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 09:41:02 -0400 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: Your message of "Mon, 07 Oct 2002 15:02:10 +0200." <3DA185D2.3010100@lemburg.com> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> Message-ID: <200210071341.g97Df2R08168@pcp02138704pcs.reston01.va.comcast.net> > > I'm still not 100% comfortable with using arbitrary coding cookies in > > the Python distribution. I like having the feature, but I would > > prefer if the Python source distribution could be viewed with tools > > that aren't encoding-aware. On my Red Hat 7.2 Linux box, I have found > > no programs that understand the UTF8 cookie. > > I just checked my SuSE 8.0 XEmacs and it seems that it doesn't > come with the UTF-8 coding system installed either. > > > They all seem to default > > to Latin-1. Since I also don't really want to assume Latin-1, I'd > > prefer if we could stick to ASCII, with the exception of tests > > explicitly needed for PEP 263. I don't want to make test_pep277.py an > > implicit PEP 263 test. > > Fine with me. I guess we ought to document this somewhere in the > Python core programming style guide (if such a document exists ;-). I've added this to PEP 8. --Guido van Rossum (home page: http://www.python.org/~guido/) From ark@research.att.com Mon Oct 7 14:47:45 2002 From: ark@research.att.com (Andrew Koenig) Date: 07 Oct 2002 09:47:45 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210070139.g971dPs19874@oma.cosc.canterbury.ac.nz> References: <200210070139.g971dPs19874@oma.cosc.canterbury.ac.nz> Message-ID: Greg> If rationals and slice objects were the same thing, Greg> it wouldn't matter! Andrew> But then how would you distinguish between x[1:4] and x[2:8]? Greg> Well, if you reduced the rational to lowest terms you Greg> would have to keep the original values around for when Greg> you were using it as a slice object... In that case, wouldn't you have a situation where a==b, a and b are the same type, but x[a]!=x[b]? -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From barry@python.org Mon Oct 7 14:59:15 2002 From: barry@python.org (Barry A. Warsaw) Date: Mon, 7 Oct 2002 09:59:15 -0400 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> Message-ID: <15777.37683.501642.428503@gargle.gargle.HOWL> >>>>> "MAL" == M writes: MAL> I just checked my SuSE 8.0 XEmacs and it seems that it MAL> doesn't come with the UTF-8 coding system installed either. Was that XEmacs built with MULE, and does that matter? (I'm struggling with utf-8 issues on the mailman-i18n lists, trying to decode and display an RFC 2047 header encoded with utf-8. Doesn't seem possible). I just visited the file in my MULE XEmacs 21.4.9 and it doesn't seem to grok the utf-8 encoding either. MAL> Fine with me. I guess we ought to document this somewhere in MAL> the Python core programming style guide (if such a document MAL> exists ;-). PEP 8 -Barry From sjoerd@acm.org Mon Oct 7 15:20:45 2002 From: sjoerd@acm.org (Sjoerd Mullender) Date: Mon, 07 Oct 2002 16:20:45 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: <15777.37683.501642.428503@gargle.gargle.HOWL> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> Message-ID: <200210071420.g97EKjw19204@indus.ins.cwi.nl> Have you seen ? I don't know whether it helps in your case, but it does tell you how to get UTF-8 support in XEmacs. On Mon, Oct 7 2002 Barry A. Warsaw wrote: > > >>>>> "MAL" == M writes: > > MAL> I just checked my SuSE 8.0 XEmacs and it seems that it > MAL> doesn't come with the UTF-8 coding system installed either. > > Was that XEmacs built with MULE, and does that matter? (I'm > struggling with utf-8 issues on the mailman-i18n lists, trying to > decode and display an RFC 2047 header encoded with utf-8. Doesn't > seem possible). I just visited the file in my MULE XEmacs 21.4.9 and > it doesn't seem to grok the utf-8 encoding either. > > MAL> Fine with me. I guess we ought to document this somewhere in > MAL> the Python core programming style guide (if such a document > MAL> exists ;-). > > PEP 8 > > -Barry > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > -- Sjoerd Mullender From mwh@python.net Mon Oct 7 15:25:01 2002 From: mwh@python.net (Michael Hudson) Date: 07 Oct 2002 15:25:01 +0100 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> Message-ID: <2m8z1az5qa.fsf@starship.python.net> barry@python.org (Barry A. Warsaw) writes: > Was that XEmacs built with MULE, and does that matter? (I'm > struggling with utf-8 issues on the mailman-i18n lists, trying to > decode and display an RFC 2047 header encoded with utf-8. Doesn't > seem possible). I just visited the file in my MULE XEmacs 21.4.9 and > it doesn't seem to grok the utf-8 encoding either. GNU Emacs 21 handles it fine (once you find a font that has all the characters). But since the coding: header idea was filched from a GNU Emacs manual, I guess that's hardly surprising... Cheers, M. -- "Well, the old ones go Mmmmmbbbbzzzzttteeeeeep as they start up and the new ones go whupwhupwhupwhooopwhooooopwhooooooommmmmmmmmm." -- Graham Reed explains subway engines on asr From tim.one@comcast.net Mon Oct 7 15:45:53 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 07 Oct 2002 10:45:53 -0400 Subject: [Python-Dev] Unclear on the way forward with unsigned integers In-Reply-To: <00ca01c26de9$4dbcedb0$e000a8c0@thomasnotebook> Message-ID: [Mark Hammond] > While this case seems quite trivial, I am starting to face this issue > more and more, especially as I am seeing these lovely "FutureWarnings" > from all my lovely 32 bit hexadecimal constants [Tim] > Sticking "L" at the end is usually all it takes. [Thomas Heller] > That removes the warnings for 'x = 0x80000000L'. What else do you think Mark may have meant by "32 bit hexadecimal constants"? > Is there a way (other than the -w command-line arg) > to suppress the warnings when doing 'hex(-1)'? The result of that is also currently platform-dependent, so you need to say what you mean in a platform-independent way. It's likely that you meant hex(-1 & 0xffffffffL) or, more directly, hex(0xffffffffL) but that someone on a 64-bit box meant something other than that. > Shouldn't there be a __future__ option? You seem to want most of all to avoid warning msgs, but __future__ options don't accomplish that: it's the *point* of __future__ thingies to spit out warnings about semantics that are going to change. I don't know that it's a useful point, though. That is, the only comments I've seen about __future__ from users in practice is from those who are annoyed by needing to say "__future__" all over the place to get at the new features they want right away. Alas, I haven't seen evidence that it eases migration. That could be because it works so well people don't feel a need to comment about it, but somehow that seems a tad unlikely . From martin@v.loewis.de Mon Oct 7 15:58:42 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 07 Oct 2002 16:58:42 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > I'm still not 100% comfortable with using arbitrary coding cookies in > the Python distribution. I like having the feature, but I would > prefer if the Python source distribution could be viewed with tools > that aren't encoding-aware. Does that mean you also want to remove test_pep263.py? > On my Red Hat 7.2 Linux box, I have found no programs that > understand the UTF8 cookie. Try IDLE :-) > They all seem to default to Latin-1. Since I also don't really want > to assume Latin-1, I'd prefer if we could stick to ASCII, with the > exception of tests explicitly needed for PEP 263. There are a number of files that have Latin-1 in them, mostly in comments. Should we change those as well? Regards, Martin From martin@v.loewis.de Mon Oct 7 16:02:46 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 07 Oct 2002 17:02:46 +0200 Subject: [Python-Dev] Unclear on the way forward with unsigned integers In-Reply-To: References: Message-ID: Tim Peters writes: > The result of that is also currently platform-dependent, so you need to say > what you mean in a platform-independent way. It's likely that you meant > > hex(-1 & 0xffffffffL) > > or, more directly, > > hex(0xffffffffL) > > but that someone on a 64-bit box meant something other than that. Alternatively, if he really means what hex(-1) will mean in the future, he should really write hex(-1L), or, if -1 is a variable, hex(long(-1)). Regards, Martin From mal@lemburg.com Mon Oct 7 16:06:56 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 07 Oct 2002 17:06:56 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> Message-ID: <3DA1A310.9030606@lemburg.com> Barry A. Warsaw wrote: >>>>>>"MAL" == M writes: > > > MAL> I just checked my SuSE 8.0 XEmacs and it seems that it > MAL> doesn't come with the UTF-8 coding system installed either. > > Was that XEmacs built with MULE, and does that matter? (I'm > struggling with utf-8 issues on the mailman-i18n lists, trying to > decode and display an RFC 2047 header encoded with utf-8. Doesn't > seem possible). I just visited the file in my MULE XEmacs 21.4.9 and > it doesn't seem to grok the utf-8 encoding either. XEmacs says: XEmacs 21.4 (patch 4) "Artificial Intelligence" (linux, Mule) of Mon Mar 25 2002 on amdsim3 So I guess, everything is in place. However the instruction posted by Sjoerd don't work for some reason. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From barry@python.org Mon Oct 7 16:20:08 2002 From: barry@python.org (Barry A. Warsaw) Date: Mon, 7 Oct 2002 11:20:08 -0400 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> Message-ID: <15777.42536.817787.472626@gargle.gargle.HOWL> >>>>> "MAL" == M writes: MAL> So I guess, everything is in place. However the instruction MAL> posted by Sjoerd don't work for some reason. I couldn't get them to work either. -Barry From sjoerd@acm.org Mon Oct 7 16:40:08 2002 From: sjoerd@acm.org (Sjoerd Mullender) Date: Mon, 07 Oct 2002 17:40:08 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: <15777.42536.817787.472626@gargle.gargle.HOWL> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> <15777.42536.817787.472626@gargle.gargle.HOWL> Message-ID: <200210071540.g97Fe9822116@indus.ins.cwi.nl> When I execute the three lisp form that were mentioned in the page I pointed you to*, my XEmacs does recognize utf-8 encoding automatically. I just find-file a file which contains UTF-8 encoded data and I see all sorts of accented characters (the ones I expect to see even!). My XEmacs version is XEmacs 21.4 (patch 6) "Common Lisp" [Lucid] (i386-redhat-linux, Mule) of Fri Feb 22 2002 on daffy.perf.redhat.com * (require 'un-define) (set-coding-priority-list '(utf-8)) (set-coding-category-system 'utf-8 'utf-8) On Mon, Oct 7 2002 Barry A. Warsaw wrote: > > >>>>> "MAL" == M writes: > > MAL> So I guess, everything is in place. However the instruction > MAL> posted by Sjoerd don't work for some reason. > I couldn't get them to work either. > -Barry > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > -- Sjoerd Mullender From martin@v.loewis.de Mon Oct 7 17:03:41 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 07 Oct 2002 18:03:41 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > On my Red Hat 7.2 Linux box, I have found no programs that > understand the UTF8 cookie. It appears that atleast kate (from KDE) can edit UTF-8 files. It won't recognize cookies, but you can specify the encoding when opening the file. Regards, Martin From martin@v.loewis.de Mon Oct 7 17:09:40 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 07 Oct 2002 18:09:40 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: <200210071540.g97Fe9822116@indus.ins.cwi.nl> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> <15777.42536.817787.472626@gargle.gargle.HOWL> <200210071540.g97Fe9822116@indus.ins.cwi.nl> Message-ID: Sjoerd Mullender writes: > When I execute the three lisp form that were mentioned in the page I > pointed you to*, my XEmacs does recognize utf-8 encoding automatically. > I just find-file a file which contains UTF-8 encoded data and I see > all sorts of accented characters (the ones I expect to see even!). [...] > (set-coding-priority-list '(utf-8)) > (set-coding-category-system 'utf-8 'utf-8) That's a slightly different issue: the question is whether it will recognize the coding of test_pep277.py even without specifying a coding priority list. Setting the priority list to utf-8 is a good thing as long is it does not exist to save all new files in UTF-8, then. Regards, Martin From barry@python.org Mon Oct 7 17:39:49 2002 From: barry@python.org (Barry A. Warsaw) Date: Mon, 7 Oct 2002 12:39:49 -0400 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> <15777.42536.817787.472626@gargle.gargle.HOWL> <200210071540.g97Fe9822116@indus.ins.cwi.nl> Message-ID: <15777.47317.362113.638027@gargle.gargle.HOWL> >>>>> "SM" == Sjoerd Mullender writes: SM> When I execute the three lisp form that were mentioned in the SM> page I pointed you to*, my XEmacs does recognize utf-8 SM> encoding automatically. Ah, I had to restart XEmacs. I added those lines to my .emacs file, restart, and visited test_pep277.py, and sure enough I see (the right) funny characters too. :) Thanks. Now I wonder if I can get utf-8 encoded RFC 2047 headers to work in email. -Barry From guido@python.org Mon Oct 7 17:43:56 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 12:43:56 -0400 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: Your message of "Mon, 07 Oct 2002 16:58:42 +0200." References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210071643.g97Ghuo16131@pcp02138704pcs.reston01.va.comcast.net> > > I'm still not 100% comfortable with using arbitrary coding cookies in > > the Python distribution. I like having the feature, but I would > > prefer if the Python source distribution could be viewed with tools > > that aren't encoding-aware. > > Does that mean you also want to remove test_pep263.py? No, see what I added to PEP 8. :-) > > On my Red Hat 7.2 Linux box, I have found no programs that > > understand the UTF8 cookie. > > Try IDLE :-) > > > They all seem to default to Latin-1. Since I also don't really want > > to assume Latin-1, I'd prefer if we could stick to ASCII, with the > > exception of tests explicitly needed for PEP 263. > > There are a number of files that have Latin-1 in them, mostly in > comments. Should we change those as well? Mostly no, see what I added to PEP 8. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Mon Oct 7 17:49:08 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 07 Oct 2002 18:49:08 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> <15777.42536.817787.472626@gargle.gargle.HOWL> <200210071540.g97Fe9822116@indus.ins.cwi.nl> Message-ID: <3DA1BB04.6070905@lemburg.com> Sjoerd Mullender wrote: > When I execute the three lisp form that were mentioned in the page I > pointed you to*, my XEmacs does recognize utf-8 encoding automatically. > I just find-file a file which contains UTF-8 encoded data and I see > all sorts of accented characters (the ones I expect to see even!). > My XEmacs version is > > XEmacs 21.4 (patch 6) "Common Lisp" [Lucid] (i386-redhat-linux, Mule) > of Fri Feb 22 2002 on daffy.perf.redhat.com > > * > (require 'un-define) > (set-coding-priority-list '(utf-8)) > (set-coding-category-system 'utf-8 'utf-8) Hmm, I get a "file not found" error. Looks like SuSE doesn't ship with the 'un-define' package. Anyway, thanks for the hint, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From barry@python.org Mon Oct 7 17:50:29 2002 From: barry@python.org (Barry A. Warsaw) Date: Mon, 7 Oct 2002 12:50:29 -0400 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> <15777.42536.817787.472626@gargle.gargle.HOWL> <200210071540.g97Fe9822116@indus.ins.cwi.nl> <3DA1BB04.6070905@lemburg.com> Message-ID: <15777.47957.552536.42555@gargle.gargle.HOWL> >>>>> "MAL" == M writes: MAL> Hmm, I get a "file not found" error. Looks like SuSE doesn't MAL> ship with the 'un-define' package. FWIW, I always build XEmacs from source --with-mule, and install all the packages from the package manager. I'm also using 21.4.9. -Barry From Gerson.Kurz@t-online.de Mon Oct 7 18:02:10 2002 From: Gerson.Kurz@t-online.de (Gerson Kurz) Date: Mon, 7 Oct 2002 19:02:10 +0200 Subject: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: <20021007133503.25994.5310.Mailman@mail.python.org> Message-ID: Hi folks, yesterday I posted a problem with importing a very large sourcefile in python 2.2.1 to clp. For the original report, see . Andrew MacIntyre suggested that 2.3 would handle this situation much better, so I got the current version from CVS, built it with cygwin and tested it. Now, instead of just a MemoryError, I get this: $ python Python 2.3a0 (#1, Oct 7 2002, 18:49:14) [GCC 2.95.3-5 (cygwin special)] on cygwin Type "help", "copyright", "credits" or "license" for more informatio >>> import test Traceback (most recent call last): Segmentation fault (core dumped) The stackdump reads Exception: STATUS_ACCESS_VIOLATION at eip=61045659 eax=1A013E68 ebx=610C02D0 ecx=610C02D0 edx=0022E6F8 esi=0022E6F8 edi=1A013E68 ebp=0022E6B0 esp=0022E698 program=C:\cygwin\usr\local\bin\python2.3.exe cs=001B ds=0023 es=0023 fs=0038 gs=0000 ss=0023 Stack trace: Frame Function Args 0022E6B0 61045659 (0022E6F8, 0022EE78, 0022F1A4, 0022F1A8) 0022E820 61048989 (1A010000, 0022ECE8, 0022EE78, 0022F1A4) 0022EE10 610462BA (00000000, 00010000, 0022EE40, 610BD2CB) 0022EE40 610096F7 (616800F4, 00000003, 0A032B14, 00000000) 0022F1F0 61071CAF (0A032B14, 00010000, 000001B6, 610AA142) 0022F210 610ACFC6 (610BF030, 0A032B14, 00010000, 000001B6) 0022F240 6109EEA8 (610BF030, 0A032B14, 6B2C525A, 6B241813) 0022F260 6109EF51 (0A032B14, 6B2C525A, 0A0113E0, 61070534) 0022FB90 6B2C52D1 (0A01D090, 0A032B14, 00000001, 0A043994) 0022FBE0 6B2C5795 (0A0358EC, 0A01D090, 0022FC40, 6B2C1861) 0022FC40 6B2C1897 (0A015DDC, 0A0229EC, 0A0358EC, 6B278D43) 0022FC70 6B2C4C8E (00000000, 0A027AA4, 0022FCC0, 6B2BEA4C) 0022FCB0 6B26BC4D (0A01BA2C, 0A027AA4, 00000000, 6B2BF4F5) 0022FCE0 6B248486 (0A01BA2C, 0A027AA4, 00000000, 6B2C4AE7) 0022FD10 6B29F082 (0A01BA2C, 0A027AA4, 00000000, 6B2C15D2) 0022FD70 6B2C15E1 (00000001, 6B2C9115, 0022FDE0, 6B2C0D25) End of stack trace (more stack frames may be present) So I guess its back in your (aka Developers) corner. You can download the source that causes this at http://p-nand-q.com/python/toobigfor2.3.zip but beware, the zipped source is 613k large (unzipped around 4.2 mb). Bye, Gerson Kurz http://p-nand-q.com From guido@python.org Mon Oct 7 18:02:44 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 13:02:44 -0400 Subject: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: Your message of "Mon, 07 Oct 2002 19:02:10 +0200." References: Message-ID: <200210071702.g97H2iw16346@pcp02138704pcs.reston01.va.comcast.net> Can you please submit a SF bug report for this? Mail to python-dev has a tendency to get lost in the pile. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Mon Oct 7 18:04:48 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 07 Oct 2002 19:04:48 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> <15777.42536.817787.472626@gargle.gargle.HOWL> <200210071540.g97Fe9822116@indus.ins.cwi.nl> <3DA1BB04.6070905@lemburg.com> <15777.47957.552536.42555@gargle.gargle.HOWL> Message-ID: <3DA1BEB0.4030306@lemburg.com> Barry A. Warsaw wrote: >>>>>>"MAL" == M writes: > > > MAL> Hmm, I get a "file not found" error. Looks like SuSE doesn't > MAL> ship with the 'un-define' package. > > FWIW, I always build XEmacs from source --with-mule, and install all > the packages from the package manager. I'm also using 21.4.9. Turns out that SuSE chose to compile XEmacs with Mule support but placed the Mule files into a separate package called 'mule-ucs-xemacs'. After I installed that package, XEmacs now correctly interprets the coding marker and works just fine with UTF-8 files. Cool :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mal@lemburg.com Mon Oct 7 18:24:28 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 07 Oct 2002 19:24:28 +0200 Subject: [Python-Dev] PEP 263 in the works (Non-ASCII characters in test_pep277.py in 2.3) References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> <15777.42536.817787.472626@gargle.gargle.HOWL> <200210071540.g97Fe9822116@indus.ins.cwi.nl> <3DA1BB04.6070905@lemburg.com> <15777.47957.552536.42555@gargle.gargle.HOWL> <3DA1BEB0.4030306@lemburg.com> Message-ID: <3DA1C34C.5090506@lemburg.com> Now that I can edit UTF-8 directly, I find a "feature" made possible by the PEP 263 support of Python 2.3 rather puzzling: Let's say I edit a file testencoding.py in XEmacs with UTF-8 support: # -*- coding: utf-8; -*- # comment =E4=F6=FC s =3D "=E4=F6=FC" u =3D u"=E4=F6=FC" print s print u.encode('latin-1') print 'works !' With Python 2.3 this prints: =C3=A4=C3=B6=C3=BC =E4=F6=FC works ! I would have expected that s turns out as "=E4=F6=FC" using print, since that's how I wrote it in the source file. This suggests to me that mixing string and Unicode literals using non-ASCII characters in a single file should probably be avoided. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Mon Oct 7 18:28:11 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 07 Oct 2002 19:28:11 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: <200210071643.g97Ghuo16131@pcp02138704pcs.reston01.va.comcast.net> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <200210071643.g97Ghuo16131@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > Mostly no, see what I added to PEP 8. :-) Ok, I've changed PEP 277 to use Unicode literals with escapes. Regards, Martin From guido@python.org Mon Oct 7 18:31:03 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 13:31:03 -0400 Subject: [Python-Dev] PEP 263 in the works (Non-ASCII characters in test_pep277.py in 2.3) In-Reply-To: Your message of "Mon, 07 Oct 2002 19:24:28 +0200." <3DA1C34C.5090506@lemburg.com> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> <15777.42536.817787.472626@gargle.gargle.HOWL> <200210071540.g97Fe9822116@indus.ins.cwi.nl> <3DA1BB04.6070905@lemburg.com> <15777.47957.552536.42555@gargle.gargle.HOWL> <3DA1BEB0.4030306@lemburg.com> <3DA1C34C.5090506@lemburg.com> Message-ID: <200210071731.g97HV3v24668@pcp02138704pcs.reston01.va.comcast.net> > Now that I can edit UTF-8 directly, I find a "feature" made > possible by the PEP 263 support of Python 2.3 rather > puzzling: > > Let's say I edit a file testencoding.py in XEmacs with UTF-8 > support: (Note that I'm viewing this as Latin-1. The comment, s and u in the source are all three the same: a-umlaut, o-umlaut, u-umlaut.) > # -*- coding: utf-8; -*- > # comment äöü > s = "äöü" > u = u"äöü" > print s > print u.encode('latin-1') > print 'works !' > > With Python 2.3 this prints: > > äöü > äöü > works ! > > I would have expected that s turns out as "äöü" using print, > since that's how I wrote it in the source file. No, because stdout isn't assumed to be UTF-8. The string s is your string encoded in UTF-8, and those are the bytes written by print. > This suggests to me that mixing string and Unicode literals > using non-ASCII characters in a single file should probably > be avoided. Or it suggests that we need a way to deal with encodings on stdout more gently. --Guido van Rossum (home page: http://www.python.org/~guido/) From zack@codesourcery.com Mon Oct 7 18:38:34 2002 From: zack@codesourcery.com (Zack Weinberg) Date: Mon, 7 Oct 2002 10:38:34 -0700 Subject: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: References: <20021007133503.25994.5310.Mailman@mail.python.org> Message-ID: <20021007173834.GA10088@codesourcery.com> On Mon, Oct 07, 2002 at 07:02:10PM +0200, Gerson Kurz wrote: > Hi folks, > > yesterday I posted a problem with importing a very large sourcefile in > python 2.2.1 to clp. For the original report, see > . Andrew MacIntyre suggested that > 2.3 would handle this situation much better, so I got the current version > from CVS, built it with cygwin and tested it. Now, instead of just a > MemoryError, I get this: For what it's worth, this may be system-specific. I was able to import your test case under both 2.2.1 and 2.3a0 with no problem, on linux. The 2.3 compiler is a lot faster, which is nice. However, running python 2.3 under valgrind produces floods of 'invalid read' and 'conditional depends on uninitialized value' errors before we ever get to the prompt, so this may well be a system-independent bug that only happens to expose itself on cygwin. zw Python 2.2.1 (#2, Sep 13 2002, 23:25:07) [GCC 2.95.4 20011002 (Debian prerelease)] on linux2 # from .py >>> profile.run('import test') 3 function calls in 49.840 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 48.980 48.980 49.830 49.830 :1(?) 1 0.010 0.010 49.840 49.840 profile:0(import test) 0 0.000 0.000 profile:0(profiler) 1 0.850 0.850 0.850 0.850 test.py:2(?) # from .pyc >>> profile.run('import test') 3 function calls in 1.140 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.420 0.420 1.130 1.130 :1(?) 1 0.010 0.010 1.140 1.140 profile:0(import test) 0 0.000 0.000 profile:0(profiler) 1 0.710 0.710 0.710 0.710 test.py:2(?) Python 2.3a0 (#30, Oct 7 2002, 10:25:24) [GCC 2.95.4 20011002 (Debian prerelease)] on linux2 # from .py >>> profile.run('import test') 3 function calls in 19.030 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 18.370 18.370 19.000 19.000 :1(?) 1 0.030 0.030 19.030 19.030 profile:0(import test) 0 0.000 0.000 profile:0(profiler) 1 0.630 0.630 0.630 0.630 test.py:2(?) # from .pyc >>> profile.run('import test') 3 function calls in 0.920 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.420 0.420 0.910 0.910 :1(?) 1 0.010 0.010 0.920 0.920 profile:0(import test) 0 0.000 0.000 profile:0(profiler) 1 0.490 0.490 0.490 0.490 test.py:2(?) From martin@v.loewis.de Mon Oct 7 18:53:03 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 07 Oct 2002 19:53:03 +0200 Subject: [Python-Dev] PEP 263 in the works (Non-ASCII characters in test_pep277.py in 2.3) In-Reply-To: <3DA1C34C.5090506@lemburg.com> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> <15777.42536.817787.472626@gargle.gargle.HOWL> <200210071540.g97Fe9822116@indus.ins.cwi.nl> <3DA1BB04.6070905@lemburg.com> <15777.47957.552536.42555@gargle.gargle.HOWL> <3DA1BEB0.4030306@lemburg.com> <3DA1C34C.5090506@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > With Python 2.3 this prints: >=20 > =C3=83=C2=A4=C3=83=C2=B6=C3=83=C2=BC As Guido explains: No, it doesn't. Your terminal display this incorrectly. Please understand that Python cannot possibly know, at run-time, what encoding a byte string has. This is only possible for Unicode strings. Regards, Martin From martin@v.loewis.de Mon Oct 7 18:57:31 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 07 Oct 2002 19:57:31 +0200 Subject: [Python-Dev] PEP 263 in the works (Non-ASCII characters in test_pep277.py in 2.3) In-Reply-To: <200210071731.g97HV3v24668@pcp02138704pcs.reston01.va.comcast.net> References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> <15777.42536.817787.472626@gargle.gargle.HOWL> <200210071540.g97Fe9822116@indus.ins.cwi.nl> <3DA1BB04.6070905@lemburg.com> <15777.47957.552536.42555@gargle.gargle.HOWL> <3DA1BEB0.4030306@lemburg.com> <3DA1C34C.5090506@lemburg.com> <200210071731.g97HV3v24668@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > Or it suggests that we need a way to deal with encodings on stdout > more gently. See python.org/sf/612627. With that patch, MAL should be able to do print u and get it out correctly no matter what terminal encoding the user uses (assuming the user follows system conventions). It would also allow to do print s.decode("utf-8").encode(sys.stdout.encoding) Either would work as long as sys.stdout is a terminal, which should include IDLE interactive sys.stdout. It is debatable whether this should work for interactive mode only, or for all scripts. Regards, Martin From Gerson.Kurz@t-online.de Mon Oct 7 19:03:56 2002 From: Gerson.Kurz@t-online.de (Gerson Kurz) Date: Mon, 7 Oct 2002 20:03:56 +0200 Subject: AW: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: <20021007173834.GA10088@codesourcery.com> Message-ID: Zack Weinberg wrote: > For what it's worth, this may be system-specific. I was able to > import your test case under both 2.2.1 and 2.3a0 with no problem, on > linux. The 2.3 compiler is a lot faster, which is nice. It seems like that. I compiled with DevStudio 6, and the import works (at least in the debug build, but thats the only one I checked). I recompiled python unter cygwin using "--with-pydebug", but to the same effect: cygwins' python.exe raises a segmentation fault. So maybe, just maybe, its a fault of cygwin ? $ uname -a CYGWIN_NT-5.0 DREIZEHN 1.3.12(0.54/3/2) 2002-07-06 02:16 i686 unknown From guido@python.org Mon Oct 7 19:03:10 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 14:03:10 -0400 Subject: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: Your message of "Mon, 07 Oct 2002 10:38:34 PDT." <20021007173834.GA10088@codesourcery.com> References: <20021007133503.25994.5310.Mailman@mail.python.org> <20021007173834.GA10088@codesourcery.com> Message-ID: <200210071803.g97I3AE08189@pcp02138704pcs.reston01.va.comcast.net> > However, running python 2.3 under valgrind produces floods of > 'invalid read' and 'conditional depends on uninitialized value' > errors before we ever get to the prompt, so this may well be a > system-independent bug that only happens to expose itself on cygwin. Are you sure these aren't false alarms? I believe I tried valgrind once on Python and every lead it gave me was a false alarm or something in libc. :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Mon Oct 7 19:09:38 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 07 Oct 2002 14:09:38 -0400 Subject: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: Message-ID: [Gerson Kurz] > ... > I compiled with DevStudio 6, and the import works (at least in the > debug build, but thats the only one I checked). I compiled > python unter cygwin using "--with-pydebug", but to the same > effect: cygwins' python.exe raises a segmentation fault. So maybe, > just maybe, its a fault of cygwin ? Dunno. If cygwin emulates the Linux malloc overcommitment policies, memory faults of this nature are unavoidable, and there's nothing we can do about it. There's something you can do about it, though: don't create 4+MB source files to begin with. I don't take problems from such tomfoolery seriously, although the younger folks here might . From pedronis@bluewin.ch Mon Oct 7 19:09:06 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 7 Oct 2002 20:09:06 +0200 Subject: [Python-Dev] segmentation fault with python2.3a0 from cvs References: Message-ID: <07c401c26e2c$a0ed0720$6d94fea9@newmexico> From: "Gerson Kurz" > $ uname -a > CYGWIN_NT-5.0 DREIZEHN 1.3.12(0.54/3/2) 2002-07-06 02:16 i686 unknown yesterday I built the Python 2.3a0 under cygwin and could rarely manage to get make test complete without some segfault. But I have not tracked the problems. I continued my hacking under Linux where I didn't get any, (apart one, but that was me :)). regards. From fdrake@acm.org Mon Oct 7 19:12:48 2002 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 7 Oct 2002 14:12:48 -0400 Subject: [Python-Dev] Python 2.2.2 beta 1 docs finalized Message-ID: <15777.52896.164933.656370@grendel.zope.com> I've posted the Python 2.2.2b1 docs online; please report any (new) problems via SourceForge with a priority of at least 6. I'd like to thank Raymond for putting so much effort into the documentation for 2.2.2; he's really doing a great job! Thanks! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mal@lemburg.com Mon Oct 7 19:19:26 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 07 Oct 2002 20:19:26 +0200 Subject: [Python-Dev] PEP 263 in the works (Non-ASCII characters in test_pep277.py in 2.3) References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <3DA13E1C.5080505@lemburg.com> <200210071228.g97CSOL06547@pcp02138704pcs.reston01.va.comcast.net> <3DA185D2.3010100@lemburg.com> <15777.37683.501642.428503@gargle.gargle.HOWL> <3DA1A310.9030606@lemburg.com> <15777.42536.817787.472626@gargle.gargle.HOWL> <200210071540.g97Fe9822116@indus.ins.cwi.nl> <3DA1BB04.6070905@lemburg.com> <15777.47957.552536.42555@gargle.gargle.HOWL> <3DA1BEB0.4030306@lemburg.com> <3DA1C34C.5090506@lemburg.com> <200210071731.g97HV3v24668@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3DA1D02E.4080304@lemburg.com> Guido van Rossum wrote: >>Now that I can edit UTF-8 directly, I find a "feature" made >>possible by the PEP 263 support of Python 2.3 rather >>puzzling: >> >>Let's say I edit a file testencoding.py in XEmacs with UTF-8 >>support: >=20 >=20 > (Note that I'm viewing this as Latin-1. The comment, s and u in the > source are all three the same: a-umlaut, o-umlaut, u-umlaut.) Oh, yes, forgot to mention that. >># -*- coding: utf-8; -*- >># comment =E4=F6=FC >>s =3D "=E4=F6=FC" >>u =3D u"=E4=F6=FC" >>print s >>print u.encode('latin-1') >>print 'works !' >> >>With Python 2.3 this prints: >> >>=C3=A4=C3=B6=C3=BC >>=E4=F6=FC >>works ! >> >>I would have expected that s turns out as "=E4=F6=FC" using print, >>since that's how I wrote it in the source file. >=20 >=20 > No, because stdout isn't assumed to be UTF-8. The string s is your > string encoded in UTF-8, and those are the bytes written by print. Uhm, I would have *expected* that behaviour -- using the left side of my head. The right side knows why this technically doesn't work out of the box :-) >>This suggests to me that mixing string and Unicode literals >>using non-ASCII characters in a single file should probably >>be avoided. >=20 > Or it suggests that we need a way to deal with encodings on stdout > more gently. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From tim.one@comcast.net Mon Oct 7 19:26:31 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 07 Oct 2002 14:26:31 -0400 Subject: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: <200210071803.g97I3AE08189@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Note that pymalloc causes valgrind to raise tons of false alarms. This was discussed previously here, but no time to track down a reference; you need a valgrind suppression file to avoid this. From skip@pobox.com Mon Oct 7 19:37:39 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 7 Oct 2002 13:37:39 -0500 Subject: [Python-Dev] tests failing in 2.2.2 In-Reply-To: <20021004135948.GA30482@epoch.metaslash.com> References: <20021004135948.GA30482@epoch.metaslash.com> Message-ID: <15777.54387.246026.24804@montanaro.dyndns.org> (a bit behind - having mail problems on my new Powerbook...) Neal> Is anyone else having problems with 2.2.2 tests? Neal> minidom & sax fail for me on Linux: ... On MacOS 10.2 I'm not getting that far. In test_builtin it gets into the list test then gobbles up all of memory. It appears the list(xrange(sys.maxint/4)) test is the culprit. This is on 2.2.2 and 2.3. Skip From Gerson.Kurz@t-online.de Mon Oct 7 19:45:14 2002 From: Gerson.Kurz@t-online.de (Gerson Kurz) Date: Mon, 7 Oct 2002 20:45:14 +0200 Subject: AW: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: Message-ID: Tim Peters wrote > There's something you can do about it, though: don't create 4+MB > source files to begin with. I don't take problems from such tomfoolery > seriously, although the younger folks here might . Ok, here is a 55 bytes sourcecode: b = {} for i in range(1000000): b[i] = [0] * 60 --> segfault with cygwin. From Gerson.Kurz@t-online.de Mon Oct 7 19:54:03 2002 From: Gerson.Kurz@t-online.de (Gerson Kurz) Date: Mon, 7 Oct 2002 20:54:03 +0200 Subject: AW: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: Message-ID: Gerson Kurz wrote > Ok, here is a 55 bytes sourcecode: > > b = {} > for i in range(1000000): > b[i] = [0] * 60 > > --> segfault with cygwin. > as a follow up, here is an interesting behaviour when used with 2.3: Python 2.3a0 (#29, Oct 7 2002, 19:54:53) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> b = {} [3319 refs] >>> for i in range(1000000): ... b[i]=[i]*60 ... [62003414 refs] >>> for k in range(1000000): ... del b[k] ... [1003419 refs] >>> print b[0] Traceback (most recent call last): File "", line 1, in ? KeyError: 0 [1003458 refs] >>> print len(b.keys()) 0 [1003621 refs] The funny thing is, the memory allocated by python_d.exe *INCREASES* when I do the del b[k]-bit thingy. OK, so I guess its time to do a gc: >>> import gc [1003753 refs] >>> gc.collect() 0 [1003758 refs] >>> gc.isenabled() 1 [1003758 refs] >>> Which frees up, like, nothing. Hm. When I do that with PythonWin 2.2.1 (#34, Apr 15 2002, 09:51:39) [MSC 32 bit (Intel)] on win32. ... >>> b = {} >>> for i in range(100000): ... b[i] = [0] * 60 ... >>> for k in range(100000): ... del b[k] ... >>> (higher numbers throw MemoryException) I get a much more expected behaviour, del b[k] frees up about 50% of the memory used (50 meg total), although gc.collect() likewise has no measurable effect. From guido@python.org Mon Oct 7 19:49:47 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 14:49:47 -0400 Subject: [Python-Dev] tests failing in 2.2.2 In-Reply-To: Your message of "Mon, 07 Oct 2002 13:37:39 CDT." <15777.54387.246026.24804@montanaro.dyndns.org> References: <20021004135948.GA30482@epoch.metaslash.com> <15777.54387.246026.24804@montanaro.dyndns.org> Message-ID: <200210071849.g97Inlt08581@pcp02138704pcs.reston01.va.comcast.net> > On MacOS 10.2 I'm not getting that far. In test_builtin it gets into the > list test then gobbles up all of memory. It appears the > list(xrange(sys.maxint/4)) test is the culprit. This is on 2.2.2 and 2.3. That's strange -- that test is specifically designed to fail before it ever gets to allocating memory. Can you track this down with a debugger? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 7 19:50:35 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 14:50:35 -0400 Subject: AW: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: Your message of "Mon, 07 Oct 2002 20:45:14 +0200." References: Message-ID: <200210071850.g97IoZa08599@pcp02138704pcs.reston01.va.comcast.net> > Ok, here is a 55 bytes sourcecode: > > b = {} > for i in range(1000000): > b[i] = [0] * 60 > > --> segfault with cygwin. Looks like a bug in CYGWIN malloc then. --Guido van Rossum (home page: http://www.python.org/~guido/) From sholden@holdenweb.com Mon Oct 7 19:48:33 2002 From: sholden@holdenweb.com (Steve Holden) Date: Mon, 7 Oct 2002 14:48:33 -0400 Subject: [Python-Dev] segmentation fault with python2.3a0 from cvs References: Message-ID: <060b01c26e32$25a166f0$6300000a@holdenweb.com> Gerson Kurz wrote ... > Tim Peters wrote > > There's something you can do about it, though: don't create 4+MB > > source files to begin with. I don't take problems from such tomfoolery > > seriously, although the younger folks here might . > > Ok, here is a 55 bytes sourcecode: > > b = {} > for i in range(1000000): > b[i] = [0] * 60 > > --> segfault with cygwin. > This is a fixed bug, due to a problem with malloc on the Cygwin platform, I believe. From what I can remember it's been fixed, but it wasn't directly a Python bug to start with. regards ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Previous .sig file retired to www.homeforoldsigs.com ----------------------------------------------------------------------- From guido@python.org Mon Oct 7 19:53:22 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 14:53:22 -0400 Subject: AW: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: Your message of "Mon, 07 Oct 2002 20:54:03 +0200." References: Message-ID: <200210071853.g97IrMO08662@pcp02138704pcs.reston01.va.comcast.net> > as a follow up, here is an interesting behaviour when used with 2.3: > > Python 2.3a0 (#29, Oct 7 2002, 19:54:53) [MSC 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> b = {} > [3319 refs] > >>> for i in range(1000000): > ... b[i]=[i]*60 > ... > [62003414 refs] > >>> for k in range(1000000): > ... del b[k] > ... > [1003419 refs] > >>> print b[0] > Traceback (most recent call last): > File "", line 1, in ? > KeyError: 0 > [1003458 refs] > >>> print len(b.keys()) > 0 > [1003621 refs] > > The funny thing is, the memory allocated by python_d.exe *INCREASES* when I > do the del b[k]-bit thingy. OK, so I guess its time to do a gc: No, there are no cycles in the data structures you create here. I suspect another CYGWIN malloc bug. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Mon Oct 7 19:53:42 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 07 Oct 2002 14:53:42 -0400 Subject: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: Message-ID: [Gerson Kurz] > Ok, here is a 55 bytes sourcecode: > > b = {} > for i in range(1000000): > b[i] = [0] * 60 > > --> segfault with cygwin. As Guido said, bug reports belong on SourceForge. It doesn't segfault for me (MSVC6 + Win98SE), although it takes a long time to complete and thrashes to disk horribly. But distinctions like that are lost forever unless they're attached to a bug report. BTW, if you need to do things "like this" frequently, buy a 64-bit box, a terabyte of RAM, and get a real operating system . From Gerson.Kurz@t-online.de Mon Oct 7 20:00:35 2002 From: Gerson.Kurz@t-online.de (Gerson Kurz) Date: Mon, 7 Oct 2002 21:00:35 +0200 Subject: AW: AW: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: <200210071853.g97IrMO08662@pcp02138704pcs.reston01.va.comcast.net> Message-ID: > No, there are no cycles in the data structures you create here. > > I suspect another CYGWIN malloc bug. > No, that was done with the DevStudio build. Note also this: Python 2.3a0 (#29, Oct 7 2002, 19:54:53) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. python_d mem usage : 2.340 K >>> b = {} [3319 refs] >>> for i in range(1000000): ... b[i] = [i] * 60 ... [62003414 refs] python_d mem usage : 376.320 K >>> del b [3413 refs] >>> python_d mem usage : 351.716 K Again, with the devstudio build (the cygwin build will crash in the loop) From zack@codesourcery.com Mon Oct 7 20:05:47 2002 From: zack@codesourcery.com (Zack Weinberg) Date: Mon, 7 Oct 2002 12:05:47 -0700 Subject: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: References: <200210071803.g97I3AE08189@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021007190547.GA18632@codesourcery.com> On Mon, Oct 07, 2002 at 02:26:31PM -0400, Tim Peters wrote: > Note that pymalloc causes valgrind to raise tons of false alarms. This was > discussed previously here, but no time to track down a reference; you need a > valgrind suppression file to avoid this. Yeah, many of them appeared to be pymalloc-related. I didn't take the time to dig through the list to see if any weren't; I wanted to raise a potential concern rather than an actual one. zw From thomas@xs4all.net Mon Oct 7 20:06:31 2002 From: thomas@xs4all.net (Thomas Wouters) Date: Mon, 7 Oct 2002 21:06:31 +0200 Subject: [Python-Dev] Unclear on the way forward with unsigned integers In-Reply-To: References: <00ca01c26de9$4dbcedb0$e000a8c0@thomasnotebook> Message-ID: <20021007190631.GD5116@xs4all.nl> On Mon, Oct 07, 2002 at 10:45:53AM -0400, Tim Peters wrote: > You seem to want most of all to avoid warning msgs, but __future__ options > don't accomplish that: it's the *point* of __future__ thingies to spit out > warnings about semantics that are going to change. > I don't know that it's a useful point, though. That is, the only comments > I've seen about __future__ from users in practice is from those who are > annoyed by needing to say "__future__" all over the place to get at the new > features they want right away. Alas, I haven't seen evidence that it eases > migration. That could be because it works so well people don't feel a need > to comment about it, but somehow that seems a tad unlikely . Well, I can tell you that __future__ has definately worked. Without it, I would not have been able to upgrade Python on many of our servers -- and because we don't need newer versions ourselves, they would not likely be installed at all. As it is, I can safely upgrade to the next major version when they come out, instead of having to start a lengthy and very energy-consuming customer-acceptance process. And this anal attitude towards compatibility isn't just me, it's company policy ;) I suspect all companies that maintain python on behalf of third parties are grateful of future. Or at least, the good ones, the others merely _should_ be grateful :) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! From tim.one@comcast.net Mon Oct 7 20:07:30 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 07 Oct 2002 15:07:30 -0400 Subject: AW: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: <200210071853.g97IrMO08662@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Gerson Kurz] > Python 2.3a0 (#29, Oct 7 2002, 19:54:53) [MSC 32 bit (Intel)] on win32 > b = {} > > [3319 refs] > > >>> for i in range(1000000): > > ... b[i]=[i]*60 > > ... > > [62003414 refs] > > >>> for k in range(1000000): > > ... del b[k] > > ... > > [1003419 refs] > > >>> print b[0] > > Traceback (most recent call last): > > File "", line 1, in ? > > KeyError: 0 > > [1003458 refs] > > >>> print len(b.keys()) > > 0 > > [1003621 refs] > > > > The funny thing is, the memory allocated by python_d.exe > *INCREASES* when I > > do the del b[k]-bit thingy. OK, so I guess its time to do a gc: [Guido] > No, there are no cycles in the data structures you create here. > > I suspect another CYGWIN malloc bug. Unless I'm reading it wrong, Gerson wasn't using Cygwin there. The int free list is immortal and unbounded, and the structure of the loop interleaves int allocations with list allocs and dict resizings. So it's unlikely memory will be "given back to the system" no matter which platform this is done on -- it's too fragmented. It doesn't make sense that sysreftotal isn't falling, though. It's as if the int free list were holding on to references ... From guido@python.org Mon Oct 7 20:11:23 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 15:11:23 -0400 Subject: AW: AW: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: Your message of "Mon, 07 Oct 2002 21:00:35 +0200." References: Message-ID: <200210071911.g97JBN008783@pcp02138704pcs.reston01.va.comcast.net> > No, that was done with the DevStudio build. Note also this: > > Python 2.3a0 (#29, Oct 7 2002, 19:54:53) [MSC 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > > python_d mem usage : 2.340 K > > >>> b = {} > [3319 refs] > >>> for i in range(1000000): > ... b[i] = [i] * 60 > ... > [62003414 refs] > > python_d mem usage : 376.320 K > > >>> del b > [3413 refs] > >>> > > python_d mem usage : 351.716 K > > Again, with the devstudio build (the cygwin build will crash in the loop) It's not a bug if freed memory is not given back to the system. Fragmentation probably prevents this in this case. --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Mon Oct 7 20:22:12 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 07 Oct 2002 21:22:12 +0200 Subject: [Python-Dev] tests failing in 2.2.2 In-Reply-To: <200210071849.g97Inlt08581@pcp02138704pcs.reston01.va.comcast.net> References: <20021004135948.GA30482@epoch.metaslash.com> <15777.54387.246026.24804@montanaro.dyndns.org> <200210071849.g97Inlt08581@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > That's strange -- that test is specifically designed to fail before it > ever gets to allocating memory. Can you track this down with a > debugger? Notice that this test does not strictly achieve that: with a four-byte pointer, you attempt to allocate 0x20000000L items (due to roundupsize), which is small enough to try a realloc. In turn, it tries to allocate 0x80000000L bytes, i.e. 2GB. On a 32-bit machine, it is possible to allocate that much memory. Regards, Martin From tim.one@comcast.net Mon Oct 7 20:22:42 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 07 Oct 2002 15:22:42 -0400 Subject: AW: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: Message-ID: [Tim] > It doesn't make sense that sysreftotal isn't falling, though. It's as > if the int free list were holding on to references ... It makes sense after all -- they're all references to the dummy key value we plug into a dict slot when del'ing a key. From guido@python.org Mon Oct 7 20:29:59 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 15:29:59 -0400 Subject: AW: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: Your message of "Mon, 07 Oct 2002 15:07:30 EDT." References: Message-ID: <200210071930.g97JU0G11290@pcp02138704pcs.reston01.va.comcast.net> > [Gerson Kurz] > > > Python 2.3a0 (#29, Oct 7 2002, 19:54:53) [MSC 32 bit (Intel)] on win32 > > b = {} > > > [3319 refs] > > > >>> for i in range(1000000): > > > ... b[i]=[i]*60 > > > ... > > > [62003414 refs] > > > >>> for k in range(1000000): > > > ... del b[k] > > > ... > > > [1003419 refs] > > > >>> print b[0] > > > Traceback (most recent call last): > > > File "", line 1, in ? > > > KeyError: 0 > > > [1003458 refs] > > > >>> print len(b.keys()) > > > 0 > > > [1003621 refs] > > > > > > The funny thing is, the memory allocated by python_d.exe > > *INCREASES* when I > > > do the del b[k]-bit thingy. OK, so I guess its time to do a gc: > > [Guido] > > No, there are no cycles in the data structures you create here. > > > > I suspect another CYGWIN malloc bug. > > Unless I'm reading it wrong, Gerson wasn't using Cygwin there. The int free > list is immortal and unbounded, and the structure of the loop interleaves > int allocations with list allocs and dict resizings. So it's unlikely > memory will be "given back to the system" no matter which platform this is > done on -- it's too fragmented. > > It doesn't make sense that sysreftotal isn't falling, though. It's as if > the int free list were holding on to references ... I tried this with smaller numbers on Linux, and it's pretty much the same (some of the [startup...] messages are from my $PYTHONSTARTUP): Python 2.2.2b1 (#21, Oct 7 2002, 15:15:32) [GCC 2.96 20000731 (Red Hat Linux 7.2 2.96-108.7.2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. [startup.py ...] [no rl completer: No module named readline] [startup.py done] >>> b = {} [16738 refs] >>> for i in range(100000): b[i]=[i]*60 [6216740 refs] >>> [6216740 refs] >>> for k in range(100000): del b[k] [116742 refs] >>> b {} [116749 refs] >>> 1 1 [116749 refs] >>> del b [16748 refs] >>> ^D [16748 refs] [exithandler()] [5704 refs] The extra references must be refs to the dummy value that the dict implementation inserts when it deletes a value. So this is not something to worry about. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Mon Oct 7 20:42:14 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 07 Oct 2002 15:42:14 -0400 Subject: [Python-Dev] Unclear on the way forward with unsigned integers In-Reply-To: <20021007190631.GD5116@xs4all.nl> Message-ID: [Thomas Wouters] > Well, I can tell you that __future__ has definately worked. Without > it, I would not have been able to upgrade Python on many of our > servers -- and because we don't need newer versions ourselves, they > would not likely be installed at all. As it is, I can safely upgrade > to the next major version when they come out, instead of having to > start a lengthy and very energy-consuming customer-acceptance process. > And this anal attitude towards compatibility isn't just me, it's > company policy ;) > > I suspect all companies that maintain python on behalf of third parties > are grateful of future. Or at least, the good ones, the others merely > _should_ be grateful :) I'm not sure this counts as a success: it doesn't sound like __future__ is helping you migrate, it sounds like __future__ is helping you *not* migrate so far. What are you going to do when 2.3 comes out, and the things that *were* future in 2.2 are no longer choices? Some code will break if you move to 2.3. Will you be able to get away with saying "but Python 2.2 warned you about this all along, so we're moving to 2.3 regardless"? If so, then I'd count it a success. From thomas.heller@ion-tof.com Mon Oct 7 21:03:56 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon, 7 Oct 2002 22:03:56 +0200 Subject: [Python-Dev] Unclear on the way forward with unsigned integers References: Message-ID: <053501c26e3c$ab4871e0$e000a8c0@thomasnotebook> > [Mark Hammond] > > While this case seems quite trivial, I am starting to face this issue > > more and more, especially as I am seeing these lovely "FutureWarnings" > > from all my lovely 32 bit hexadecimal constants > > [Tim] > > Sticking "L" at the end is usually all it takes. > > [Thomas Heller] > > That removes the warnings for 'x = 0x80000000L'. > > What else do you think Mark may have meant by "32 bit hexadecimal > constants"? > > > Is there a way (other than the -w command-line arg) > > to suppress the warnings when doing 'hex(-1)'? > > The result of that is also currently platform-dependent, so you need to say > what you mean in a platform-independent way. It's likely that you meant > > hex(-1 & 0xffffffffL) > > or, more directly, > > hex(0xffffffffL) > > but that someone on a 64-bit box meant something other than that. > > > Shouldn't there be a __future__ option? > > You seem to want most of all to avoid warning msgs, but __future__ options > don't accomplish that: it's the *point* of __future__ thingies to spit out > warnings about semantics that are going to change. Yes, I want to avoid the warnings. I've reread pep237 (is this the relevant one?) again. Do I understand this correctly, that all the following expressions will be 'true' and will compile/execute without any warnings (on a 32bit box): Python 2.2: 0x80000000 == -2147483648 Python 2.3: 0x80000000L == 2147483648 Python 2.x (Phase B1?): 0x80000000L == 2147483648 Python 2.y (Phase B2 and later): 0x80000000 == 2147483648 Regards, Thomas From guido@python.org Mon Oct 7 21:12:17 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 16:12:17 -0400 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: Your message of "Sun, 06 Oct 2002 13:13:30 EDT." References: Message-ID: <200210072012.g97KCIS11560@pcp02138704pcs.reston01.va.comcast.net> > Python 2.2.2 beta 1 is coming out, and I was hoping to float a final > request to have the fix for bug #537450 applied to the 2.2 maint > branch (it is a trivial backport, since the 2.3 patch applies just > fine). A back-port was originally passed over, because it changes > semantics from Python 2.2.1. I argue that the changed semantics are > desirable and constitute a clear bug fix, which closes holes that > can allow very obscure errors. It would be very surprising to me if > this change breaks any extant code, since it is virtually impossible > to take meaningful or safe advantage of the original semantics. [...] > I assert that no sensible developer would ever write code that > relied on the pre-2.3 CVS semantics. I've done a survey of much of > the meta-class code posted to or discussed on Python-dev and > comp.lang.python, and none rely on this behavior. Plus, even if > they did, we have already decided they will have to change it when > 2.3 comes out. > > If you are not entirely convinced by the above argument, please > consider a compromise solution: put the fix in for 2.2.1 Beta 1. If > anyone has complaints based on extant code, then we can always back > it out for Beta 2, or the final release. OK, that's a deal. It's in 2.2.2b1. If there are problems with it I may back it out before 2.2.2final though. --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs@penguin.theopalgroup.com Mon Oct 7 21:18:05 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 7 Oct 2002 16:18:05 -0400 (EDT) Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: <200210072012.g97KCIS11560@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Mon, 7 Oct 2002, Guido van Rossum wrote: > > If you are not entirely convinced by the above argument, please > > consider a compromise solution: put the fix in for 2.2.1 Beta 1. If > > anyone has complaints based on extant code, then we can always back > > it out for Beta 2, or the final release. > > OK, that's a deal. It's in 2.2.2b1. If there are problems with it I > may back it out before 2.2.2final though. Thanks -- I appreciate it. This will indeed make life easier for me. I'll bet you a beer that nobody (who doesn't read this list) will complain. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From guido@python.org Mon Oct 7 21:32:42 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 16:32:42 -0400 Subject: [Python-Dev] Unclear on the way forward with unsigned integers In-Reply-To: Your message of "Mon, 07 Oct 2002 22:03:56 +0200." <053501c26e3c$ab4871e0$e000a8c0@thomasnotebook> References: <053501c26e3c$ab4871e0$e000a8c0@thomasnotebook> Message-ID: <200210072032.g97KWgA11625@pcp02138704pcs.reston01.va.comcast.net> > I've reread pep237 (is this the relevant one?) again. Yes. > Do I understand this correctly, that all the following expressions > will be 'true' and will compile/execute without any warnings (on > a 32bit box): > > Python 2.2: > 0x80000000 == -2147483648 Yes. > Python 2.3: > 0x80000000L == 2147483648 Also in 2.2. Before that, 0x80000000L had the same value, but 2147483648 raised OverflowError. > Python 2.x (Phase B1?): > 0x80000000L == 2147483648 > > Python 2.y (Phase B2 and later): > 0x80000000 == 2147483648 Yes. --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas@xs4all.net Mon Oct 7 22:22:03 2002 From: thomas@xs4all.net (Thomas Wouters) Date: Mon, 7 Oct 2002 23:22:03 +0200 Subject: [Python-Dev] Unclear on the way forward with unsigned integers In-Reply-To: References: <20021007190631.GD5116@xs4all.nl> Message-ID: <20021007212203.GR621@xs4all.nl> On Mon, Oct 07, 2002 at 03:42:14PM -0400, Tim Peters wrote: > [Thomas Wouters] > > Well, I can tell you that __future__ has definately worked. Without > > it, I would not have been able to upgrade Python on many of our > > servers -- and because we don't need newer versions ourselves, they > > would not likely be installed at all. As it is, I can safely upgrade > > to the next major version when they come out, instead of having to > > start a lengthy and very energy-consuming customer-acceptance process. > > And this anal attitude towards compatibility isn't just me, it's > > company policy ;) > I'm not sure this counts as a success: it doesn't sound like __future__ is > helping you migrate, it sounds like __future__ is helping you *not* migrate > so far. What are you going to do when 2.3 comes out, and the things that > *were* future in 2.2 are no longer choices? Some code will break if you > move to 2.3. Will you be able to get away with saying "but Python 2.2 > warned you about this all along, so we're moving to 2.3 regardless"? If so, > then I'd count it a success. We've already been there (2.0 -> 2.1 -> 2.2) and that's exactly what we did. Customers had several months, some more, to notice warnings about deprecated features and the like, and to use the __future__ import if they wanted newer features. That gives us a very justified 'too bad' if they come complaining afterwards anyway. It also inspires more trust from the more, uhm, 'traditional' companies if you have a clear and documented upgrade path for software. They often have to hire outside programmers to do the work, and they want to know about upkeep as well. I could talk for hours about how we are switching from an old hysterical-raisin[*] setup that was hard to update and practically impossible to upgrade major portions of, where it would take months to migrate customers from BSDI 3.1 to BSDI 4.0, to our current 'ideal' setup where everything is maintained on a small set of 'master' servers that distribute their changes to hundreds of other servers automatically (which www.python.org is part of, by the way) -- but I won't bore you with the details. Rest assured, __future__ imports (or at least 'no breakage without a warning one major release before', which doesn't strictly require __future__) make it a heck of a lot easier. Backwards-doesn't-mean-retarded'ly y'rs, [*] For some reason the Jargon file lists this as 'hysterical reasons', in case you're wondering what it is. Must be a hysterical raisin. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! From guido@python.org Mon Oct 7 22:45:59 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 17:45:59 -0400 Subject: [Python-Dev] Python 2.2.2b1 checkin freeze Message-ID: <200210072145.g97LjxG12739@pcp02138704pcs.reston01.va.comcast.net> I've committed the last changes for Python 2.2.2b1 -- the Misc/NEWS file. I'm now going to follow the rest of the PEP 102 script until the release is done. In the mean time, please no more checkins to the 2.2 branch! --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Mon Oct 7 22:54:15 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 7 Oct 2002 16:54:15 -0500 Subject: [Python-Dev] Warning about use of long double - what's the correct approach? Message-ID: <15778.647.849183.55911@montanaro.dyndns.org> When building from CVS head, I get this warning for each file compiled: gcc -c -DNDEBUG -g -I. -I../Include -DPy_BUILD_CORE -o Objects/rangeobject.o ../Objects/rangeobject.c In file included from ../Include/Python.h:70, from ../Objects/rangeobject.c:3: ../Include/objimpl.h:252: warning: use of `long double' type; its size may change in a future release ../Include/objimpl.h:252: warning: (Long double usage is reported only once for each file. ../Include/objimpl.h:252: warning: To disable this warning, use -Wno-long-double.) The culprit is the long double alignment field in: /* GC information is stored BEFORE the object structure. */ typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; int gc_refs; } gc; long double dummy; /* force worst-case alignment */ } PyGC_Head; I don't know what the best way to fix this is. Should I look at making configure add -Wno-long-double in the proper situations or should that alignment field be something else in situations where the use of long double would generate a warning? Skip From tim.one@comcast.net Mon Oct 7 23:02:37 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 07 Oct 2002 18:02:37 -0400 Subject: [Python-Dev] Unclear on the way forward with unsigned integers In-Reply-To: <20021007212203.GR621@xs4all.nl> Message-ID: [Thomas Wouters] > We've already been there (2.0 -> 2.1 -> 2.2) and that's exactly > what we did. Cool! Then it's exactly as I expected: it works so well, nobody thought to mention it . > Customers had several months, some more, to notice warnings about > deprecated features and the like, and to use the __future__ import > if they wanted newer features. That gives us a very justified 'too > bad' if they come complaining afterwards anyway. That was all the intent. > It also inspires more trust from the more, uhm, 'traditional' > companies if you have a clear and documented upgrade path for > software. Let's quit here while we're ahead <0.9 wink>. From greg@cosc.canterbury.ac.nz Mon Oct 7 23:38:29 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 08 Oct 2002 11:38:29 +1300 (NZDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Message-ID: <200210072238.g97McTk06649@oma.cosc.canterbury.ac.nz> Andrew> In that case, wouldn't you have a situation where a==b, a and Andrew> b are the same type, but x[a]!=x[b]? Yes. Not such a good idea, I suppose. I've just had another idea: {2/3} This can't be confused with a dict, because there's no ":" in it. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From dave@boost-consulting.com Mon Oct 7 23:56:09 2002 From: dave@boost-consulting.com (David Abrahams) Date: 07 Oct 2002 18:56:09 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210072238.g97McTk06649@oma.cosc.canterbury.ac.nz> References: <200210072238.g97McTk06649@oma.cosc.canterbury.ac.nz> Message-ID: Greg Ewing writes: > Andrew> In that case, wouldn't you have a situation where a==b, a and > Andrew> b are the same type, but x[a]!=x[b]? > > Yes. Not such a good idea, I suppose. > > I've just had another idea: > > {2/3} 2r/3 would be nicer IMO. -- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From guido@python.org Tue Oct 8 00:05:11 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 19:05:11 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Tue, 08 Oct 2002 11:38:29 +1300." <200210072238.g97McTk06649@oma.cosc.canterbury.ac.nz> References: <200210072238.g97McTk06649@oma.cosc.canterbury.ac.nz> Message-ID: <200210072305.g97N5Bq20760@pcp02138704pcs.reston01.va.comcast.net> > I've just had another idea: > > {2/3} > > This can't be confused with a dict, because there's > no ":" in it. On the slight chance that you're serious, I'd like to reserve {...} without colons for set notation, see PEP 218. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Tue Oct 8 00:39:49 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 7 Oct 2002 18:39:49 -0500 Subject: [Python-Dev] Re: [snake-farm] Autodetect /usr/ccs/bin? In-Reply-To: <200210061329.g96DTOs14718@pcp02138704pcs.reston01.va.comcast.net> References: <20021006115713.GB22400@lysator.liu.se> <200210061329.g96DTOs14718@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15778.6981.362515.852029@montanaro.dyndns.org> >> Should the configure script try with /usr/ccs/bin/ar and ld if none >> is explicitly supplied? I think you should add /usr/ccs/{lib,bin,include} and /sw/{lib,bin,include} to places you automatically search for "useful stuff". As Anders pointed out, /usr/ccs is something of a standard among the more proprietary of the Unix platforms. /sw/... is where the fink gang installs all the various GNU goodies on MacOS X. While you won't need to search /sw/bin for ar and ld, you might well find useful third-party libraries there which various wrapper modules can use. When I configured XEmacs without asking it to look in /sw/... it came up with a fairly feeble configuration (no jpeg or png support). That said, perhaps the /sw thing will work with a simple change to setup.py. I'm testing a change right now to see if my build picks up readline or not. Skip From skip@pobox.com Tue Oct 8 00:56:23 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 7 Oct 2002 18:56:23 -0500 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> References: <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15778.7975.178103.577482@montanaro.dyndns.org> Guido> BTW, the Wiki is here: http://manatee.mojam.com/py222wiki/ Guido> I tried to set up an email subscription but the Wiki isn't Guido> configured to support that... That would be my mistake. I didn't even know you could do that. Under the rubric of "better late than never" I'll see what I can do to enable whatever email subscription capabilities MoinMoin supports. Skip From jacobs@penguin.theopalgroup.com Tue Oct 8 01:02:35 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 7 Oct 2002 20:02:35 -0400 (EDT) Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: <15778.7975.178103.577482@montanaro.dyndns.org> Message-ID: On Mon, 7 Oct 2002, Skip Montanaro wrote: > Guido> BTW, the Wiki is here: http://manatee.mojam.com/py222wiki/ > > Guido> I tried to set up an email subscription but the Wiki isn't > Guido> configured to support that... > > That would be my mistake. I didn't even know you could do that. Under the > rubric of "better late than never" I'll see what I can do to enable whatever > email subscription capabilities MoinMoin supports. On a somewhat tangental note, MoinMoin breaks with Python 2.3 CVS -- it expects integers from some of the os/filesystem time routines were changed to return floats. The fix is trivial, of course, but it is interesting to note where real code breaks. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From greg@cosc.canterbury.ac.nz Tue Oct 8 01:18:10 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 08 Oct 2002 13:18:10 +1300 (NZDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210072305.g97N5Bq20760@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> Guido: > On the slight chance that you're serious, I'd like to reserve {...} > without colons for set notation, see PEP 218. I have to agree that would be a better use for them. So, I'm now reduced to suggesting <2/3> Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tdelaney@avaya.com Tue Oct 8 01:18:16 2002 From: tdelaney@avaya.com (Delaney, Timothy) Date: Tue, 8 Oct 2002 10:18:16 +1000 Subject: [Python-Dev] [win32] Killing MSVC's _alloca Message-ID: > From: Guido van Rossum [mailto:guido@python.org] > > I don't understand. How many Unix emulations on Windows do we need? > We've already absorbed a sheer endless set of patches to make it work > on CYGWIN. How does Mingw differ? None at all. MinGW is not a Unix emulation. There is a minimal shell environment (MSYS) which works with MinGW (allowing use of the autoconf/make toolchain) but the produced binaries have no reliance on anything except the native win32 libraries. I was recently trying to use MinGW to compile Python CVS at home - I don't own, not will I pay for MSVC++. I occasionally have to use it at work, but only from the command line ;) I gave up when it got into posixmodule.c and there was lots of stuff there I didn't know about. I personally am very pleased that Gerhard is working so hard to make this a viable compilation option on Windows. http://www.mingw.org/ http://www.mingw.org/msys.shtml Tim Delaney From brian@sweetapp.com Tue Oct 8 01:32:39 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Mon, 07 Oct 2002 17:32:39 -0700 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> Message-ID: <002301c26e62$353cc2a0$df7e4e18@brianspiv1700> Greg Ewing wrote: > I have to agree that would be a better use for them. > > So, I'm now reduced to suggesting > > <2/3> Is that going to be easy to parse? >>> 1 <5/2> -3 1 Cheers, Brian From tdelaney@avaya.com Tue Oct 8 01:39:58 2002 From: tdelaney@avaya.com (Delaney, Timothy) Date: Tue, 8 Oct 2002 10:39:58 +1000 Subject: [Python-Dev] test_pep277 vs Win98 Message-ID: > From: martin@v.loewis.de [mailto:martin@v.loewis.de] > > Tim Peters writes: > > > > Furthermore, how would I implement > os.path.supports_unicode_filenames > > > for ntpath? It seems that something must be exported from os. > > > > This sounds like, for now, whether a platform supports this > or not is a > > matter of hand-forcing a config symbol, where only a few > Windows flavors are > > currently justified in saying "yup, works here". > > Repeating my question: How can I hand-force a config symbol to True on > NT+ and to False on W9x? Sounds like an exceptional situation to me. try: try to use problem filename(s) yup_works_here = True except IOError: # or whatever is appropriate yup_works_here = False The test cases then simply test that it works for all cases Tim Delaney From guido@python.org Tue Oct 8 01:59:59 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 20:59:59 -0400 Subject: [Python-Dev] RELEASED: Python 2.2.2b1 Message-ID: <200210080100.g98104x22506@pcp02138704pcs.reston01.va.comcast.net> I'm happy to announce the release of Python 2.2.2 beta 1. Python 2.2.2 will be a fully backwards compatible bugfix release for Python 2.2.1. I'm shooting for a short beta test cycle: hopefully I can make the final 2.2.2 release next week with minimal changes from 2.2.2b1. Python 2.2.2 has a large number of bug fixes in all areas of the interpreter and the library; for a full account see the release notes, available via the link below. The new release is available here: http://www.python.org/2.2.2/ As usual, a Windows installer and a Unix/Linux source tarball are made available, as well as tarballs of the documentation in various forms. At the moment, no Mac version is available, though I expect that Jack Jansen will start working on one soon. Likewise, no Linux RPMs are available; I expect that Sean Reifschneider will prepare Linux RPMs for 2.2.2 final. I'm especially grateful to Michael Hudson, Raymond Hettinger and Neal Norwitz for their hard work on backporting the large volume of patches from Python 2.3. Without them this release would not have been possible. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Oct 8 02:08:12 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 21:08:12 -0400 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: Your message of "Mon, 07 Oct 2002 20:02:35 EDT." References: Message-ID: <200210080108.g9818CJ22598@pcp02138704pcs.reston01.va.comcast.net> > On a somewhat tangental note, MoinMoin breaks with Python 2.3 CVS -- > it expects integers from some of the os/filesystem time routines > were changed to return floats. The fix is trivial, of course, but > it is interesting to note where real code breaks. I wonder if perhaps we should be more conservative here and stick to ints. This was discussed here and we didn't know of any software that would break. Now we do... --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Tue Oct 8 02:12:06 2002 From: barry@python.org (Barry A. Warsaw) Date: Mon, 7 Oct 2002 21:12:06 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <200210072305.g97N5Bq20760@pcp02138704pcs.reston01.va.comcast.net> <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> Message-ID: <15778.12518.911824.400047@gargle.gargle.HOWL> >>>>> "GE" =3D=3D Greg Ewing writes: GE> I have to agree that would be a better use for them. GE> So, I'm now reduced to suggesting GE> <2/3> Everyone's missing the obvious: 2=F73 :) -Barry From guido@python.org Tue Oct 8 02:12:53 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 21:12:53 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Mon, 07 Oct 2002 18:56:09 EDT." References: <200210072238.g97McTk06649@oma.cosc.canterbury.ac.nz> Message-ID: <200210080112.g981Crr22726@pcp02138704pcs.reston01.va.comcast.net> > 2r/3 would be nicer IMO. Or 2/3r. (The r binds only to the 3, but of course the binary operator rules kick in with the same effect.) Frankly, this is the only sane notation for rationals I've seen so far in this discussion. --Guido van Rossum (home page: http://www.python.org/~guido/) From greg@cosc.canterbury.ac.nz Tue Oct 8 02:23:44 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 08 Oct 2002 14:23:44 +1300 (NZDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <002301c26e62$353cc2a0$df7e4e18@brianspiv1700> Message-ID: <200210080123.g981Nie07039@oma.cosc.canterbury.ac.nz> Brian Quinlan : > Is that going to be easy to parse? > > >>> 1 <5/2> -3 > 1 Yes, but >>> 1 + <5/2> * -3 File "", line 1 1 + <5/2> * -3 ^ SyntaxError: invalid syntax When used as a rational literal, the <...> will always have some operator separating it from any adjacent expression. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg@cosc.canterbury.ac.nz Tue Oct 8 02:25:11 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 08 Oct 2002 14:25:11 +1300 (NZDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <15778.12518.911824.400047@gargle.gargle.HOWL> Message-ID: <200210080125.g981PBC07059@oma.cosc.canterbury.ac.nz> barry@python.org (Barry A. Warsaw): > Everyone's missing the obvious: >=20 > 2=F73 >=20 > :) Well, I was trying to limit myself to ASCII characters. If/when we get Unicode source, though... Greg Ewing, Computer Science Dept, +---------------------------------= -----+ University of Canterbury,=09 | A citizen of NewZealandCorp, a=09 | Christchurch, New Zealand=09 | wholly-owned subsidiary of USA Inc. = | greg@cosc.canterbury.ac.nz=09 +------------------------------------= --+ From dave@boost-consulting.com Tue Oct 8 02:20:39 2002 From: dave@boost-consulting.com (David Abrahams) Date: 07 Oct 2002 21:20:39 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210080112.g981Crr22726@pcp02138704pcs.reston01.va.comcast.net> References: <200210072238.g97McTk06649@oma.cosc.canterbury.ac.nz> <200210080112.g981Crr22726@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > > 2r/3 would be nicer IMO. > > Or 2/3r. (The r binds only to the 3, but of course the binary > operator rules kick in with the same effect.) > > Frankly, this is the only sane notation for rationals I've seen so far > in this discussion. I liked 2r/3 because it gives the sense that r/ is the rational division operator, where // is the whatever-the-hell-it-is division operator. I don't know if it works in the grammar to be able to say x r/ y though. Does it? -- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From greg@cosc.canterbury.ac.nz Tue Oct 8 02:27:50 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 08 Oct 2002 14:27:50 +1300 (NZDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210080112.g981Crr22726@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210080127.g981Ro807070@oma.cosc.canterbury.ac.nz> Guido: > Or 2/3r. (The r binds only to the 3, but of course the binary > operator rules kick in with the same effect.) That's actually not too bad! The only trouble is I keep wanting to pronounce it as "two-thirds raw"... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido@python.org Tue Oct 8 02:37:58 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 07 Oct 2002 21:37:58 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Mon, 07 Oct 2002 21:20:39 EDT." References: <200210072238.g97McTk06649@oma.cosc.canterbury.ac.nz> <200210080112.g981Crr22726@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210080137.g981bwU23754@pcp02138704pcs.reston01.va.comcast.net> > I liked 2r/3 because it gives the sense that r/ is the rational > division operator, where // is the whatever-the-hell-it-is division > operator. I don't know if it works in the grammar to be able to say > > x r/ y > > though. Does it? That would require changes to the tokenizer. But I am against r/ on different grounds: it's not the kind of grouping of symbols that one would expect. People are used to 12L, 1j and then it's a small step to 2r. There were also precedents for r"..." and u"...": C's w"...". If you want a precedent for 2/, you'd have to search in Lisp or Forth or other (nearly) grammar-less languages. --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs@penguin.theopalgroup.com Tue Oct 8 02:49:55 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 7 Oct 2002 21:49:55 -0400 (EDT) Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: <200210080108.g9818CJ22598@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Mon, 7 Oct 2002, Guido van Rossum wrote: > > On a somewhat tangental note, MoinMoin breaks with Python 2.3 CVS -- > > it expects integers from some of the os/filesystem time routines > > were changed to return floats. The fix is trivial, of course, but > > it is interesting to note where real code breaks. > > I wonder if perhaps we should be more conservative here and stick to > ints. This was discussed here and we didn't know of any software that > would break. Now we do... I don't really care much either way, and I suspect that the author(s) of MoinMoin don't either. I can see the theoretical benefit of floating point timestamps, though I don't really need them (yet). Once I do (if ever), then I'll definitly be stumping for them. On the other hand, the code that broke in MoinMoin was ugly, and deserved what it got. So maybe I do care a little... +1 for leaving it in +1 for getting some sleep since I'm clearly starting to hallucinate -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From dave@boost-consulting.com Tue Oct 8 03:29:21 2002 From: dave@boost-consulting.com (David Abrahams) Date: 07 Oct 2002 22:29:21 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210080137.g981bwU23754@pcp02138704pcs.reston01.va.comcast.net> References: <200210072238.g97McTk06649@oma.cosc.canterbury.ac.nz> <200210080112.g981Crr22726@pcp02138704pcs.reston01.va.comcast.net> <200210080137.g981bwU23754@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > > I liked 2r/3 because it gives the sense that r/ is the rational > > division operator, where // is the whatever-the-hell-it-is division > > operator. I don't know if it works in the grammar to be able to say > > > > x r/ y > > > > though. Does it? > > That would require changes to the tokenizer. > > But I am against r/ on different grounds: it's not the kind of > grouping of symbols that one would expect. People are used to 12L, 1j > and then it's a small step to 2r. You're right. And now that I look at it, if 2r is a rational with value 2, and if you can divide ints by rationals, then 1/2r makes a lot of sense. I wasn't looking at it that way (but I am now, and liking it). > There were also precedents for > r"..." and u"...": C's w"...". If you want a precedent for 2/, you'd > have to search in Lisp or Forth or other (nearly) grammar-less > languages. Oh, please, don't remind me about those funky Forth symbols. I guess it has less to do with grammar than with lexemes, though. -- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From tim.one@comcast.net Tue Oct 8 03:51:55 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 07 Oct 2002 22:51:55 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210080112.g981Crr22726@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Noting that Scheme has two sets of optional numeric literal prefixes: #b #o #d #h number is binary, octal, decimal (the default), or hex #e #i number is exact or inexact This avoids conflating exactness with representation, and, e.g., #e#b1.001 is exactly 9/8 (although that Dr. Scheme allows a radix point in a binary literal appears to be an extension of the Scheme std). By default, a numeric literal is inexect iff it contains a radix point or an exponent, but #e or #i can override that. > (exact? 2) #t > (exact? #i2) #f > (exact? 6.02e23) #f > (exact? #e6.02e23) #t I think this works very well. The same rule about default exactness would be appropriate for Python too, and an r suffix meaning what a #e prefix means in Scheme would be a fine idea by my lights (the effect of an #i prefix can be gotten via including a decimal point for decimal literals, and inexact literals in other bases are rarely useful). From Raymond Hettinger" Message-ID: <005301c26e81$de469780$870fa044@oemcomputer> Traceback (most recent call last): File "idle.py", line 3, in ? import PyShell File "C:\PYTHON22\Tools\idle\PyShell.py", line 16, in ? from EditorWindow import EditorWindow, fixwordbreaks File "C:\PYTHON22\Tools\idle\EditorWindow.py", line 13, in ? from IdleConf import idleconf File "C:\PYTHON22\Tools\idle\IdleConf.py", line 5, in ? from ConfigParser import ConfigParser, NoOptionError, NoSectionError File "C:\PYTHON22\lib\ConfigParser.py", line 172, in ? class ConfigParser: File "C:\PYTHON22\lib\ConfigParser.py", line 308, in ConfigParser _boolean_states = {'1': True, 'yes': True, 'true': True, 'on': True, NameError: name 'True' is not defined From Raymond Hettinger" <005301c26e81$de469780$870fa044@oemcomputer> Message-ID: <005b01c26e83$19ac7960$870fa044@oemcomputer> It looks like the windows build includes an updated python_d.dll but leaves an out-of-date python.dll: PYTHON22 DLL 843,833 05-08-02 2:34a python22.dll PYTHON~1 DLL 1,744,943 08-02-02 8:55p python22_d.dll Raymond Hettinger ----- Original Message ----- From: "Raymond Hettinger" To: ; "Guido van Rossum" Sent: Tuesday, October 08, 2002 12:19 AM Subject: [Python-Dev] RELEASED: Python 2.2.2b1 -- Arrrggggggghhhh! > Traceback (most recent call last): > File "idle.py", line 3, in ? > import PyShell > File "C:\PYTHON22\Tools\idle\PyShell.py", line 16, in ? > from EditorWindow import EditorWindow, fixwordbreaks > File "C:\PYTHON22\Tools\idle\EditorWindow.py", line 13, in ? > from IdleConf import idleconf > File "C:\PYTHON22\Tools\idle\IdleConf.py", line 5, in ? > from ConfigParser import ConfigParser, NoOptionError, NoSectionError > File "C:\PYTHON22\lib\ConfigParser.py", line 172, in ? > class ConfigParser: > File "C:\PYTHON22\lib\ConfigParser.py", line 308, in ConfigParser > _boolean_states = {'1': True, 'yes': True, 'true': True, 'on': True, > NameError: name 'True' is not defined > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > From martin@v.loewis.de Tue Oct 8 07:53:14 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 08 Oct 2002 08:53:14 +0200 Subject: [Python-Dev] Warning about use of long double - what's the correct approach? In-Reply-To: <15778.647.849183.55911@montanaro.dyndns.org> References: <15778.647.849183.55911@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > When building from CVS head, I get this warning for each file compiled: > > gcc -c -DNDEBUG -g -I. -I../Include -DPy_BUILD_CORE -o Objects/rangeobject.o ../Objects/rangeobject.c [...] > I don't know what the best way to fix this is. Should I look at making > configure add -Wno-long-double in the proper situations or should that > alignment field be something else in situations where the use of long double > would generate a warning? As a starting point, explain the problem :-) What architecture, what compiler, and why is sizeof long double going to change? Regards, Martin From mhammond@skippinet.com.au Tue Oct 8 08:22:23 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Tue, 8 Oct 2002 17:22:23 +1000 Subject: [Python-Dev] Snapshot win32all builds of interest? In-Reply-To: <200209301500.g8UF0mL18039@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Me] > > I'm wondering if there is any interest in me making regular, > > basically untested win32all builds against the current Python CVS > > tree? [Guido] > I think that would be useful, especially if the current 2.2-compatible > win32all does not work with 2.3, or if you have added features since > that was last released. OK - first such snapshot done. See the link from starship.python.net/crew/mhammond This version installs no shortcuts, does not compile .pyc files etc - you are pretty much on your own. Pythonwin\start_pythonwin.pyw is installed to start Pythonwin, but you must do so manually. Please let me know if you try it. Mark. From martin@v.loewis.de Tue Oct 8 08:42:40 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 08 Oct 2002 09:42:40 +0200 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: <200210080108.g9818CJ22598@pcp02138704pcs.reston01.va.comcast.net> References: <200210080108.g9818CJ22598@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > I wonder if perhaps we should be more conservative here and stick to > ints. This was discussed here and we didn't know of any software that > would break. Now we do... What convinced me the last time is that time stamps were already floats on the Mac. Regards, Martin From martin@v.loewis.de Tue Oct 8 08:44:39 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 08 Oct 2002 09:44:39 +0200 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: References: Message-ID: "Delaney, Timothy" writes: > None at all. MinGW is not a Unix emulation. There is a minimal shell > environment (MSYS) which works with MinGW (allowing use of the autoconf/make > toolchain) but the produced binaries have no reliance on anything except the > native win32 libraries. To me, this is a clear indication that there is a bug in mingw here. If they want to be a Windows compiler, they have to provide MSVC compatibility (and they do, to some degree); this also means they have to provide _alloca, since this is the Windows name of that feature. Regards, Martin From mwh@python.net Tue Oct 8 10:58:26 2002 From: mwh@python.net (Michael Hudson) Date: 08 Oct 2002 10:58:26 +0100 Subject: [Python-Dev] Warning about use of long double - what's the correct approach? In-Reply-To: Skip Montanaro's message of "Mon, 7 Oct 2002 16:54:15 -0500" References: <15778.647.849183.55911@montanaro.dyndns.org> Message-ID: <2m4rbxntfh.fsf@starship.python.net> Skip Montanaro writes: > When building from CVS head, I get this warning for each file compiled: On Mac OS X, right? > gcc -c -DNDEBUG -g -I. -I../Include -DPy_BUILD_CORE -o Objects/rangeobject.o ../Objects/rangeobject.c > In file included from ../Include/Python.h:70, > from ../Objects/rangeobject.c:3: > ../Include/objimpl.h:252: warning: use of `long double' type; its size may change in a future release > ../Include/objimpl.h:252: warning: (Long double usage is reported only once for each file. > ../Include/objimpl.h:252: warning: To disable this warning, use -Wno-long-double.) > > The culprit is the long double alignment field in: > > /* GC information is stored BEFORE the object structure. */ > typedef union _gc_head { > struct { > union _gc_head *gc_next; > union _gc_head *gc_prev; > int gc_refs; > } gc; > long double dummy; /* force worst-case alignment */ > } PyGC_Head; > > I don't know what the best way to fix this is. Should I look at making > configure add -Wno-long-double in the proper situations or should that > alignment field be something else in situations where the use of long double > would generate a warning? I thought that's what configure already did: case $ac_sys_system in Darwin*) OPT="$OPT -Wno-long-double -no-cpp-precomp";; esac If I've guessed wrong and you're not on OS X, give us some help :) Cheers, M. -- I'm sorry, was my bias showing again? :-) -- William Tanksley, 13 May 2000 From mwh@python.net Tue Oct 8 11:02:09 2002 From: mwh@python.net (Michael Hudson) Date: 08 Oct 2002 11:02:09 +0100 Subject: [Python-Dev] Re: [snake-farm] Autodetect /usr/ccs/bin? In-Reply-To: Skip Montanaro's message of "Mon, 7 Oct 2002 18:39:49 -0500" References: <20021006115713.GB22400@lysator.liu.se> <200210061329.g96DTOs14718@pcp02138704pcs.reston01.va.comcast.net> <15778.6981.362515.852029@montanaro.dyndns.org> Message-ID: <2m1y71nt9a.fsf@starship.python.net> Skip Montanaro writes: > >> Should the configure script try with /usr/ccs/bin/ar and ld if none > >> is explicitly supplied? > > I think you should add /usr/ccs/{lib,bin,include} and /sw/{lib,bin,include} > to places you automatically search for "useful stuff". As Anders pointed > out, /usr/ccs is something of a standard among the more proprietary of the > Unix platforms. /sw/... is where the fink gang installs all the various GNU > goodies on MacOS X. While you won't need to search /sw/bin for ar and ld, > you might well find useful third-party libraries there which various wrapper > modules can use. When I configured XEmacs without asking it to look in > /sw/... it came up with a fairly feeble configuration (no jpeg or png > support). That said, perhaps the /sw thing will work with a simple change > to setup.py. I'm testing a change right now to see if my build picks up > readline or not. If this is your change: Index: setup.py =================================================================== RCS file: /cvsroot/python/python/dist/src/setup.py,v retrieving revision 1.111 diff -c -r1.111 setup.py *** setup.py 25 Sep 2002 14:59:57 -0000 1.111 --- setup.py 8 Oct 2002 10:01:00 -0000 *************** *** 229,234 **** --- 229,236 ---- # Ensure that /usr/local is always used add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib') add_dir_to_list(self.compiler.include_dirs, '/usr/local/include') + add_dir_to_list(self.compiler.library_dirs, '/sw/lib') + add_dir_to_list(self.compiler.include_dirs, '/sw/include') if os.path.normpath(sys.prefix) != '/usr': add_dir_to_list(self.compiler.library_dirs, It will. This is so easy, I don't think it's worth bothering with in the core... Cheers, M. -- Considering that this thread is completely on-topic in the way only c.l.py threads can be, I think I can say that you should replace "Oblivion" with "Gravity", and increase your Radiohead quotient. -- Ben Wolfson, comp.lang.python From hughett@mercur.uphs.upenn.edu Tue Oct 8 13:41:31 2002 From: hughett@mercur.uphs.upenn.edu (Paul Hughett) Date: Tue, 8 Oct 2002 08:41:31 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> (message from Greg Ewing on Tue, 08 Oct 2002 13:18:10 +1300 (NZDT)) References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> Message-ID: <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> Greg Ewing wrote: > So, I'm now reduced to suggesting > <2/3> How about 2r3, which could be pronounced "2 rational 3" and is syntactically very similar to the universal 2e3? Paul Hughett From mal@lemburg.com Tue Oct 8 13:51:49 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 08 Oct 2002 14:51:49 +0200 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> Message-ID: <3DA2D4E5.3020301@lemburg.com> Paul Hughett wrote: > Greg Ewing wrote: > > >>So, I'm now reduced to suggesting > > >> <2/3> > > > How about 2r3, which could be pronounced "2 rational 3" and is syntactically > very similar to the universal 2e3? What's so hard about rat(2,3) ? And if that doesn't look right, simply do: R = rat(1,1) 2/R/3 This works without any changes to the language. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From Norman Vine" Message-ID: "Martin v. Loewis" wrote in message news:m3bs65xtlk.fsf@mira.informatik.hu-berlin.de... > "Delaney, Timothy" writes: > > > None at all. MinGW is not a Unix emulation. There is a minimal shell > > environment (MSYS) which works with MinGW (allowing use of the autoconf/make > > toolchain) but the produced binaries have no reliance on anything except the > > native win32 libraries. > > To me, this is a clear indication that there is a bug in mingw > here. If they want to be a Windows compiler, they have to provide MSVC > compatibility (and they do, to some degree); this also means they have > to provide _alloca, since this is the Windows name of that feature. #ifdef __MINGW32__ #ifndef _alloca #define _alloca(x) __builtin_alloca(x) #endif #endif HTH Norman From guido@python.org Tue Oct 8 14:01:16 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 09:01:16 -0400 Subject: [Python-Dev] RELEASED: Python 2.2.2b1 -- Arrrggggggghhhh! In-Reply-To: Your message of "Tue, 08 Oct 2002 00:28:05 EDT." <005b01c26e83$19ac7960$870fa044@oemcomputer> References: <200210080100.g98104x22506@pcp02138704pcs.reston01.va.comcast.net> <005301c26e81$de469780$870fa044@oemcomputer> <005b01c26e83$19ac7960$870fa044@oemcomputer> Message-ID: <200210081301.g98D1H725141@pcp02138704pcs.reston01.va.comcast.net> > It looks like the windows build includes an updated > python_d.dll but leaves an out-of-date python.dll: > > PYTHON22 DLL 843,833 05-08-02 2:34a python22.dll > PYTHON~1 DLL 1,744,943 08-02-02 8:55p python22_d.dll Neither of those was created by the installer (look at the dates!). Perhaps somehow the installer couldn't write your python.dll because it was in use? The python22_d.dll is never written by the installer -- we don't distribute any debug-mode files. I'm sure the installer is fine, the problem must be in your setup. --Guido van Rossum (home page: http://www.python.org/~guido/) From list-python@ccraig.org Tue Oct 8 14:28:27 2002 From: list-python@ccraig.org (Christopher A. Craig) Date: 08 Oct 2002 09:28:27 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <3DA2D4E5.3020301@lemburg.com> References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> <3DA2D4E5.3020301@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > What's so hard about rat(2,3) ? > > And if that doesn't look right, simply do: > > R = rat(1,1) > > 2/R/3 > > This works without any changes to the language. My thoughts are moving towards that. I also wouldn't especially mind seeing rationals added to the standard library instead of the core, though I'm not sure how that effects Numeric. The two best suggestions (IMHO) I've seen so far are the above and Guido's 2/3r where that is actually the int 2 over the rational 3r. -- Christopher A. Craig "Going to school make a person educated, any more than going to a garage makes a person a car" Slashdot From mwh@python.net Tue Oct 8 14:36:28 2002 From: mwh@python.net (Michael Hudson) Date: 08 Oct 2002 14:36:28 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib/distutils sysconfig.py,1.44,1.44.6.1 In-Reply-To: mwh@users.sourceforge.net's message of "Mon, 07 Oct 2002 03:38:35 -0700" References: Message-ID: <2m4rbxrr1f.fsf@starship.python.net> mwh@users.sourceforge.net writes: > Index: sysconfig.py > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Lib/distutils/sysconfig.py,v > retrieving revision 1.44 > retrieving revision 1.44.6.1 > diff -C2 -d -r1.44 -r1.44.6.1 > *** sysconfig.py 6 Dec 2001 20:51:35 -0000 1.44 > --- sysconfig.py 7 Oct 2002 10:38:33 -0000 1.44.6.1 > *************** > *** 24,40 **** > EXEC_PREFIX = os.path.normpath(sys.exec_prefix) > > ! # Boolean; if it's true, we're still building Python, so > ! # we use different (hard-wired) directories. > ! > ! python_build = 0 > ! > ! def set_python_build(): > ! """Set the python_build flag to true. > > ! This means that we're building Python itself. Only called from > ! the setup.py script shipped with Python. > ! """ > ! global python_build > python_build = 1 Hmm, I wonder if I should put a do-nothing set_python_build back in for 2.2.2. Does anyone call this from their build scripts? I guess it's possible... Cheers, M. -- GET *BONK* BACK *BONK* IN *BONK* THERE *BONK* -- Naich using the troll hammer in cam.misc From thomas@xs4all.net Tue Oct 8 14:46:10 2002 From: thomas@xs4all.net (Thomas Wouters) Date: Tue, 8 Oct 2002 15:46:10 +0200 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <3DA2D4E5.3020301@lemburg.com> References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> <3DA2D4E5.3020301@lemburg.com> Message-ID: <20021008134610.GE5116@xs4all.nl> On Tue, Oct 08, 2002 at 02:51:49PM +0200, M.-A. Lemburg wrote: > What's so hard about rat(2,3) ? I have to agree with MAL. I know, rationally, why a rational literal (or is it literal rational ?) is desirable, but it feels like clutter and all of the proposed syntactic solutions strike me as bad ideas. I'd much rather have the above (which deals with everyone else's favorite data type and their requests to have builtin support for it at the same time) than a not-quite-perfect way to spell a rational literally. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! From thomas@xs4all.net Tue Oct 8 14:55:49 2002 From: thomas@xs4all.net (Thomas Wouters) Date: Tue, 8 Oct 2002 15:55:49 +0200 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> Message-ID: <20021008135549.GF5116@xs4all.nl> On Tue, Oct 08, 2002 at 08:41:31AM -0400, Paul Hughett wrote: > Greg Ewing wrote: > > So, I'm now reduced to suggesting > > > <2/3> > How about 2r3, which could be pronounced "2 rational 3" and is syntactically > very similar to the universal 2e3? 2e3 may be universal, I wouldn't say it's very universally used. I personally never use the e-notation, and I can't say I find it very readable. Who do we expect to use rationals ? Is it something we want newbies to learn ? Will high/grade-school students with not compsci/math background to recognize the e-notation be using rationals ? I'm not going all happy-dayzy about rationals in the first place, but I know that is because I understand the implementation and tradeoffs of floating point, and they are what I want :) I'd also say rational-literals are not that important. Looking at my own Python code, I very rarely need a floating-point literal to start with. Strings, plenty, dicts and lists fairly often, integers every now and then, but floating point numbers very rarely, and almost all of them are just '0.0' or an integer expressed as float to force float-division. Most of my float objects come from (library) functions that return them. If we expect newbies to prefer rationals over fp, we need syntax that is clear to them (which may very well be just 'rat()'.) If we expect hard-core mathers like Tim and Moshe and Chris Tismer and even Guido to use them, I'd say we don't need syntax support for them and can just live with rat(). And I have to say some of the proposals for literal rationals I saw were very disturbingly sick. Some of you need help, and some need to spend less time using XML :-) Beautiful-is-better-than-ug'ly y'rs, -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! From neal@metaslash.com Tue Oct 8 14:58:32 2002 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 8 Oct 2002 09:58:32 -0400 Subject: [Python-Dev] Python 2.2.2b1 checkin freeze In-Reply-To: <200210072145.g97LjxG12739@pcp02138704pcs.reston01.va.comcast.net> References: <200210072145.g97LjxG12739@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021008135832.GA22587@epoch.metaslash.com> On Mon, Oct 07, 2002 at 05:45:59PM -0400, Guido van Rossum wrote: > I've committed the last changes for Python 2.2.2b1 -- the Misc/NEWS > file. I'm now going to follow the rest of the PEP 102 script until > the release is done. In the mean time, please no more checkins to the > 2.2 branch! I know of 2 problems. Both are simple and low-risk to fix. In NEWS, alrady is misspelled. tstate->tick_counter is not initialized in Python/pystate.c line 138. If I remember correct, there is some inconsistent use/doc of True/False vs 1/0 in ConfigParser.py. I don't think this is worth fixing in 2.2.2. I valgrind'ed 2.2.2 on Sept 26 and all was clean. Neal From skip@pobox.com Tue Oct 8 14:49:18 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 8 Oct 2002 08:49:18 -0500 Subject: [Python-Dev] Warning about use of long double - what's the correct approach? In-Reply-To: References: <15778.647.849183.55911@montanaro.dyndns.org> Message-ID: <15778.57950.590904.335471@montanaro.dyndns.org> >>>>> "Martin" == Martin v Loewis writes: Martin> Skip Montanaro writes: >> When building from CVS head, I get this warning for each file compiled: Martin> As a starting point, explain the problem :-) What architecture, Martin> what compiler, and why is sizeof long double going to change? Sorry. MacOS 10.2. Apple's gcc version of 3.1. I have no idea why it's going to change. I'm just the messenger. ;-) Skip From guido@python.org Tue Oct 8 15:43:27 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 10:43:27 -0400 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib/distutils sysconfig.py,1.44,1.44.6.1 In-Reply-To: Your message of "Tue, 08 Oct 2002 14:36:28 BST." <2m4rbxrr1f.fsf@starship.python.net> References: <2m4rbxrr1f.fsf@starship.python.net> Message-ID: <200210081443.g98EhRj12368@odiug.zope.com> > Hmm, I wonder if I should put a do-nothing set_python_build back in > for 2.2.2. Does anyone call this from their build scripts? I guess > it's possible... Good idea. better safe than sorry. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Tue Oct 8 15:45:02 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 8 Oct 2002 10:45:02 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> <3DA2D4E5.3020301@lemburg.com> <20021008134610.GE5116@xs4all.nl> Message-ID: <15778.61294.171669.963839@gargle.gargle.HOWL> >>>>> "TW" == Thomas Wouters writes: TW> I have to agree with MAL. I know, rationally, why a rational TW> literal (or is it literal rational ?) is desirable, but it TW> feels like clutter and all of the proposed syntactic solutions TW> strike me as bad ideas. I'd much rather have the above (which TW> deals with everyone else's favorite data type and their TW> requests to have builtin support for it at the same time) than TW> a not-quite-perfect way to spell a rational literally. It seems to me that Python has a tradition of deferring syntax decisions until way after the more important issues have been worked out. Perhaps we should do the same here, IOW, get the rational library into the core and see if a rational literal makes that big a difference for readability or maintainability. It may not, but then at least we'll still have rationals. -Barry From jeremy@alum.mit.edu Tue Oct 8 16:16:51 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Tue, 8 Oct 2002 11:16:51 -0400 Subject: [Python-Dev] Python 2.2.2b1 checkin freeze In-Reply-To: <20021008135832.GA22587@epoch.metaslash.com> References: <200210072145.g97LjxG12739@pcp02138704pcs.reston01.va.comcast.net> <20021008135832.GA22587@epoch.metaslash.com> Message-ID: <15778.63203.959350.840343@slothrop.zope.com> >>>>> "NN" == Neal Norwitz writes: NN> I valgrind'ed 2.2.2 on Sept 26 and all was clean. Shouldn't that be valground? Jeremy From hughett@mercur.uphs.upenn.edu Tue Oct 8 15:56:34 2002 From: hughett@mercur.uphs.upenn.edu (Paul Hughett) Date: Tue, 8 Oct 2002 10:56:34 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021008135549.GF5116@xs4all.nl> (message from Thomas Wouters on Tue, 8 Oct 2002 15:55:49 +0200) References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> <20021008135549.GF5116@xs4all.nl> Message-ID: <200210081456.g98EuYx24034@mercur.uphs.upenn.edu> Paul Hughett wrote: >> How about 2r3, which could be pronounced "2 rational 3" and is syntactically >> very similar to the universal 2e3? Thomas Wouters wrote: > 2e3 may be universal, I wouldn't say it's very universally used. I > personally never use the e-notation, and I can't say I find it very >readable I think I was a bit misleading there. I mentioned the similarity to 2e3 as an argument that it would be easy to modify the lexer and parser to handle 2r3. I doubt that we want to inflict that analogy on the learner; if nothing else, they'd keep trying to interpret the 3 as an exponent. > I'd also say rational-literals are not that important. This may be the key decision. Do we really need rational literals, or can we live with constructors? Using rat(2,3) seems fine to me as the constructor, and I'm not all that enamoured of rational literals. On the other hand, I would hardly ever use rationals; I regard them as a specialized tool akin to symbolic algebra (to which they are arguably essential). If we do want rational literals, well then 2r3 seems a reasonably painless way to get them. > If we expect newbies to prefer rationals over fp... I don't. I'd say the right thing for newbies is decimal floating point, rather than rationals. Not that I object to adding rationals to Python; they're a specialized tool, but a powerful one within their niche. Paul Hughett From thomas@xs4all.net Tue Oct 8 15:56:30 2002 From: thomas@xs4all.net (Thomas Wouters) Date: Tue, 8 Oct 2002 16:56:30 +0200 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <15778.61294.171669.963839@gargle.gargle.HOWL> References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> <3DA2D4E5.3020301@lemburg.com> <20021008134610.GE5116@xs4all.nl> <15778.61294.171669.963839@gargle.gargle.HOWL> Message-ID: <20021008145630.GX621@xs4all.nl> On Tue, Oct 08, 2002 at 10:45:02AM -0400, Barry A. Warsaw wrote: > It seems to me that Python has a tradition of deferring syntax > decisions until way after the more important issues have been > worked out. I'd agree, except that I don't think there are any issues more important than syntax :) I know what you mean, though, and I do agree with that. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! From guido@python.org Tue Oct 8 15:52:22 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 10:52:22 -0400 Subject: [Python-Dev] Python 2.2.2b1 checkin freeze In-Reply-To: Your message of "Tue, 08 Oct 2002 09:58:32 EDT." <20021008135832.GA22587@epoch.metaslash.com> References: <200210072145.g97LjxG12739@pcp02138704pcs.reston01.va.comcast.net> <20021008135832.GA22587@epoch.metaslash.com> Message-ID: <200210081452.g98EqNL14643@odiug.zope.com> > I know of 2 problems. Both are simple and low-risk to fix. In NEWS, > alrady is misspelled. tstate->tick_counter is not initialized in > Python/pystate.c line 138. Fixed -- in time for the final 2.2.2 release. > If I remember correct, there is some inconsistent use/doc of > True/False vs 1/0 in ConfigParser.py. I don't think this is worth > fixing in 2.2.2. Definitely not. > I valgrind'ed 2.2.2 on Sept 26 and all was clean. Thanks! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Oct 8 16:09:36 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 11:09:36 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Mon, 07 Oct 2002 22:51:55 EDT." References: Message-ID: <200210081509.g98F9aU15528@odiug.zope.com> [Tim] > Noting that Scheme has two sets of optional numeric literal prefixes: [...] > This avoids conflating exactness with representation, and, e.g., {...] > By default, a numeric literal is inexect iff it contains a radix > point or an exponent, but #e or #i can override that. [...] > I think this works very well. The same rule about default exactness > would be appropriate for Python too, and an r suffix meaning what a > #e prefix means in Scheme would be a fine idea by my lights (the > effect of an #i prefix can be gotten via including a decimal point > for decimal literals, and inexact literals in other bases are rarely > useful). I don't think Python needs the full matrix of exact and inexact versions of all kind of numbers. I see not enough need for inexact ints or rationals, nor for exact floats or complex numbers. So I'd like to continue our partition of numeric types as follows: exact | inexact ------------------------------------ int/long rational | float complex But we can do this, which is pretty much what Tim proposes in the end: 1 -> int 1.0 -> float 1r -> rational 1.0r -> rational If we ever add a fixed-point decimal type, that could use 'f' for a suffix. If we ever add a floating-point decimal type (like the one Aahz is working on -- I mistakenly called it a fixed-point type before) then it could use 'd' as a suffix, or it could become the default and we could use 'b' as a suffix to get binary floating point. But I also agree with a recent trend in this thread: let's not rush to add syntax. Let's first add rationals to the library. I hereby declare this thread closed. (Ha, ha. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Tue Oct 8 16:10:36 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 8 Oct 2002 10:10:36 -0500 Subject: [Python-Dev] Bug #537450 Improper object initialization for type.__new__ In-Reply-To: <15778.7975.178103.577482@montanaro.dyndns.org> References: <200210062057.g96KvaG29958@pcp02138704pcs.reston01.va.comcast.net> <15778.7975.178103.577482@montanaro.dyndns.org> Message-ID: <15778.62828.970616.101696@montanaro.dyndns.org> Guido> I tried to set up an email subscription but the Wiki isn't Guido> configured to support that... Skip> That would be my mistake. Now enabled. Skip From skip@pobox.com Tue Oct 8 16:14:20 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 8 Oct 2002 10:14:20 -0500 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: References: <200210072238.g97McTk06649@oma.cosc.canterbury.ac.nz> Message-ID: <15778.63052.576188.619350@montanaro.dyndns.org> David> 2r/3 would be nicer IMO. I like this the best of those I've seen so far. I don't see any reason that rational literals need to contain a division symbol, so "2r" makes a fine rational. What about "2.5r"? Can that just be expanded to "25r/10" by the lexical analyzer? Skip From barry@python.org Tue Oct 8 16:15:19 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 8 Oct 2002 11:15:19 -0400 Subject: [Python-Dev] Python 2.2.2b1 checkin freeze References: <200210072145.g97LjxG12739@pcp02138704pcs.reston01.va.comcast.net> <20021008135832.GA22587@epoch.metaslash.com> <15778.63203.959350.840343@slothrop.zope.com> Message-ID: <15778.63111.912539.824214@gargle.gargle.HOWL> >>>>> "NN" == Neal Norwitz writes: NN> I valgrind'ed 2.2.2 on Sept 26 and all was clean. >>>>> "JH" == Jeremy Hylton writes: JH> Shouldn't that be valground? The technical term is "valgrunged". From Richie Hindle" Hi, First a brief introduction, since this is my first post to python-dev: I'm a professional software engineer, mostly using C/C++ and Python. I use Python a lot for my own projects and some of you will know me from the spambayes project, as the author of pop3proxy.py. I was interested to see Armin's patch [617312] to "allow the C profile and trace functions to alter some fields of the current PyFrameObject". I'm writing a Python debugger, and as Armin says, his patch "could also be used by advanced debuggers to allow the execution point to be modified by the user." A "Set-next-statement" feature is on my list of nice-to-have-but-hard-to-do jobs, and this patch could make it easy. (I know the blockstack presents a problem, but if it's insurmountable I'm happy to ignore that and to only allow skipping to lines in the same block - 90% of my use of the equivalent feature in MSVC is to skip back one line or forward one line.) There's one fly in the ointment - I'm trying to keep my debugger in pure Python, and Armin's patch only applies to C trace functions. If frame.f_lasti were writable by Python trace functions, pure Python debuggers (including pdb) could implement Set-Next-Statement. Here's a small script that demonstrates what I mean, and the one-line patch to frameobject.c that enables it: ----------------------------------------------------------------------- """Demonstration of debugger 'Set-next-statement' feature. Requires Python 2.2.2 with writeable frame.f_lasti. Prints "1, 2, 2, 3".""" import sys, pprint def example(): print 1, print 2, print 3 # Line 9; we skip back to 8 the first time we hit this. class DebuggerHook: def __init__(self): self.done = False # Have we done the Set-next-statement yet? self.lasti = 0 # The frame.f_lasti of the previous line. sys.settrace(self.trace) def trace(self, frame, action, arg): if not self.done and frame.f_lineno == 9: # This is the first time we've hit line 9; skip back to 8. frame.f_lasti = self.lasti self.done = True else: # Store the instruction offset of this line. self.lasti = frame.f_lasti return self.trace debugger = DebuggerHook() example() # Prints "1, 2, 2, 3" ----------------------------------------------------------------------- *** frameobject.c Tue Oct 8 08:16:39 2002 --- frameobject-222b1.c Tue Oct 8 08:16:12 2002 *************** *** 16,20 **** {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, {"f_globals", T_OBJECT, OFF(f_globals), RO}, ! {"f_lasti", T_INT, OFF(f_lasti)}, {"f_lineno", T_INT, OFF(f_lineno), RO}, {"f_restricted",T_INT, OFF(f_restricted),RO}, --- 16,20 ---- {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, {"f_globals", T_OBJECT, OFF(f_globals), RO}, ! {"f_lasti", T_INT, OFF(f_lasti), RO}, {"f_lineno", T_INT, OFF(f_lineno), RO}, {"f_restricted",T_INT, OFF(f_restricted),RO}, ----------------------------------------------------------------------- For completeness, f_lineno should probably be writable as well, but I'm keeping the changes minimal for the purposes of discussion. Is this a reasonable suggestion for 2.3, or is it giving people too much rope? A nasty consequence is that you can write Python code that causes Python to seg-fault, but you have to be doing some fairly advanced stuff for that to happen. I would say that the power of a Set-next-statement feature in pdb and other debuggers is worth the price, but others may disagree...? -- Richie Hindle richie@entrian.com From mwh@python.net Tue Oct 8 16:27:16 2002 From: mwh@python.net (Michael Hudson) Date: 08 Oct 2002 16:27:16 +0100 Subject: [Python-Dev] Set-next-statement in Python debuggers In-Reply-To: richie@entrian.com's message of "Tue, 08 Oct 2002 16:19:11 +0100" References: Message-ID: <2m4rbxne7f.fsf@starship.python.net> richie@entrian.com writes: > I was interested to see Armin's patch [617312] to "allow the C > profile and trace functions to alter some fields of the current > PyFrameObject". > > I'm writing a Python debugger, and as Armin says, his patch "could > also be used by advanced debuggers to allow the execution point to > be modified by the user." A "Set-next-statement" feature is on my > list of nice-to-have-but-hard-to-do jobs, and this patch could make > it easy. (I know the blockstack presents a problem, but if it's > insurmountable I'm happy to ignore that and to only allow skipping > to lines in the same block - 90% of my use of the equivalent feature > in MSVC is to skip back one line or forward one line.) > > There's one fly in the ointment - I'm trying to keep my debugger in > pure Python, and Armin's patch only applies to C trace functions. > If frame.f_lasti were writable by Python trace functions, pure > Python debuggers (including pdb) could implement Set-Next-Statement. > > Here's a small script that demonstrates what I mean, and the > one-line patch to frameobject.c that enables it: [...] > For completeness, f_lineno should probably be writable as well, but > I'm keeping the changes minimal for the purposes of discussion. I'd prefer a different approach; make f_lasti a descriptor so you can do some sanity checking on the values (i.e. fits in the f_code.co_code, actually is an opcode boundary and sets f_lineno). > Is this a reasonable suggestion for 2.3, or is it giving people too > much rope? A nasty consequence is that you can write Python code > that causes Python to seg-fault, but you have to be doing some > fairly advanced stuff for that to happen. > > I would say that the power of a Set-next-statement feature in pdb > and other debuggers is worth the price, but others may disagree...? Hmm. Not sure. Given all the ways Armin's just found to make the interpreter crap out, one more doesn't seem that exciting -- but perhaps this is the wrong attitude... Cheers, M. -- I'm not sure that the ability to create routing diagrams similar to pretzels with mad cow disease is actually a marketable skill. -- Steve Levin -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html From Richie Hindle" Message-ID: Hi, [Hell's bells, sorry, here's this again with the line-breaks fixed! How embarrassing...] First a brief introduction, since this is my first post to python-dev: I'm a professional software engineer, mostly using C/C++ and Python. I use Python a lot for my own projects and some of you will know me from the spambayes project, as the author of pop3proxy.py. I was interested to see Armin's patch [617312] to "allow the C profile and trace functions to alter some fields of the current PyFrameObject". I'm writing a Python debugger, and as Armin says, his patch "could also be used by advanced debuggers to allow the execution point to be modified by the user." A "Set-next-statement" feature is on my list of nice-to-have-but-hard-to-do jobs, and this patch could make it easy. (I know the blockstack presents a problem, but if it's insurmountable I'm happy to ignore that and to only allow skipping to lines in the same block - 90% of my use of the equivalent feature in MSVC is to skip back one line or forward one line.) There's one fly in the ointment - I'm trying to keep my debugger in pure Python, and Armin's patch only applies to C trace functions. If frame.f_lasti were writable by Python trace functions, pure Python debuggers (including pdb) could implement Set-Next-Statement. Here's a small script that demonstrates what I mean, and the one-line patch to frameobject.c that enables it: ----------------------------------------------------------------------- """Demonstration of debugger 'Set-next-statement' feature. Requires Python 2.2.2 with writeable frame.f_lasti. Prints "1, 2, 2, 3".""" import sys, pprint def example(): print 1, print 2, print 3 # Line 9; we skip back to 8 the first time we hit this. class DebuggerHook: def __init__(self): self.done = False # Have we done the Set-next-statement yet? self.lasti = 0 # The frame.f_lasti of the previous line. sys.settrace(self.trace) def trace(self, frame, action, arg): if not self.done and frame.f_lineno == 9: # This is the first time we've hit line 9; skip back to 8. frame.f_lasti = self.lasti self.done = True else: # Store the instruction offset of this line. self.lasti = frame.f_lasti return self.trace debugger = DebuggerHook() example() # Prints "1, 2, 2, 3" ----------------------------------------------------------------------- *** frameobject.c Tue Oct 8 08:16:39 2002 --- frameobject-222b1.c Tue Oct 8 08:16:12 2002 *************** *** 16,20 **** {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, {"f_globals", T_OBJECT, OFF(f_globals), RO}, ! {"f_lasti", T_INT, OFF(f_lasti)}, {"f_lineno", T_INT, OFF(f_lineno), RO}, {"f_restricted",T_INT, OFF(f_restricted),RO}, --- 16,20 ---- {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, {"f_globals", T_OBJECT, OFF(f_globals), RO}, ! {"f_lasti", T_INT, OFF(f_lasti), RO}, {"f_lineno", T_INT, OFF(f_lineno), RO}, {"f_restricted",T_INT, OFF(f_restricted),RO}, ----------------------------------------------------------------------- For completeness, f_lineno should probably be writable as well, but I'm keeping the changes minimal for the purposes of discussion. Is this a reasonable suggestion for 2.3, or is it giving people too much rope? A nasty consequence is that you can write Python code that causes Python to seg-fault, but you have to be doing some fairly advanced stuff for that to happen. I would say that the power of a Set-next-statement feature in pdb and other debuggers is worth the price, but others may disagree...? -- Richie Hindle richie@entrian.com From guido@python.org Tue Oct 8 16:42:18 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 11:42:18 -0400 Subject: [Python-Dev] Set-next-statement in Python debuggers In-Reply-To: Your message of "Tue, 08 Oct 2002 16:19:11 BST." References: Message-ID: <200210081542.g98FgIn15739@odiug.zope.com> > *** frameobject.c Tue Oct 8 08:16:39 2002 > --- frameobject-222b1.c Tue Oct 8 08:16:12 2002 > *************** > *** 16,20 **** > {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, > {"f_globals", T_OBJECT, OFF(f_globals), RO}, > ! {"f_lasti", T_INT, OFF(f_lasti)}, > {"f_lineno", T_INT, OFF(f_lineno), RO}, > {"f_restricted",T_INT, OFF(f_restricted),RO}, > --- 16,20 ---- > {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, > {"f_globals", T_OBJECT, OFF(f_globals), RO}, > ! {"f_lasti", T_INT, OFF(f_lasti), RO}, > {"f_lineno", T_INT, OFF(f_lineno), RO}, > {"f_restricted",T_INT, OFF(f_restricted),RO}, A backwards diff! He posted a backwards diff! Aargh! :-) > Is this a reasonable suggestion for 2.3, or is it giving people too > much rope? A nasty consequence is that you can write Python code > that causes Python to seg-fault, but you have to be doing some > fairly advanced stuff for that to happen. I would say that the > power of a Set-next-statement feature in pdb and other debuggers is > worth the price, but others may disagree...? I can't approve a Python-level feature that explicitly opens up a loophole of this kind. But if you add a C "getset" wrapper that allows setting this only to safe values (i.e. within the same block and pointing at the beginning of a bytecode) it might be deemed safe enough. It should probably silently update f_lineno as well. Spare no cost. It's up to you to write the patch, submit it to SF, and prove that it's safe. --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Tue Oct 8 16:37:46 2002 From: dave@boost-consulting.com (David Abrahams) Date: 08 Oct 2002 11:37:46 -0400 Subject: [Python-Dev] read-only properties Message-ID: I was just working on some docs for Boost.Python and thought I'd write down what happens when you try to write to a read-only property (I use properties to expose C++ data members among other things). So: >>> class X(object): ... def get(self): return 'x' ... l = property(get) ... >>> x = X() >>> x.l 'x' >>> x.l = 1 Traceback (most recent call last): File "", line 1, in ? AttributeError: can't set attribute This struck me as a very poor error message, given that mechanism in Boost.Python v1 used to say: Traceback (innermost last): File "", line 1, in ? AttributeError: 'l' attribute is read-only Is it worth doing something about this? -- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From tim@zope.com Tue Oct 8 16:55:24 2002 From: tim@zope.com (Tim Peters) Date: Tue, 8 Oct 2002 11:55:24 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210081509.g98F9aU15528@odiug.zope.com> Message-ID: [Guido] > I don't think Python needs the full matrix of exact and inexact > versions of all kind of numbers. I see not enough need for inexact > ints or rationals, nor for exact floats or complex numbers. #i3 is "an inexact integer" in Scheme, but that says nothing about how it's *implemented*. In fact, it's almost certainly implemented as a float, much as #e3.0 is almost certainly implemented as an integer. Scheme hides the internal implementation, so that the programmer doesn't need to care; the flip side is that programmers who do care can't force the issue in a portable way. > So I'd like to continue our partition of numeric types as follows: > > exact | inexact > ------------------------------------ > int/long rational | float complex > > But we can do this, which is pretty much what Tim proposes in the end: > > 1 -> int > 1.0 -> float > > 1r -> rational > 1.0r -> rational Yes, that's exactly what I was proposing. s/r/#i/ and it's the same as Scheme's rules, although Scheme doesn't promise anything about internal representation. > If we ever add a fixed-point decimal type, that could use 'f' for a > suffix. If we ever add a floating-point decimal type (like the one > Aahz is working on -- I mistakenly called it a fixed-point type > before) then it could use 'd' as a suffix, or it could become the > default and we could use 'b' as a suffix to get binary floating point. Etc. > But I also agree with a recent trend in this thread: let's not rush > to add syntax. Let's first add rationals to the library. > > I hereby declare this thread closed. (Ha, ha. :-) You should have put that sentence first -- my editor still doesn't go backwards . From skip@pobox.com Tue Oct 8 16:59:21 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 8 Oct 2002 10:59:21 -0500 Subject: [Python-Dev] Re: [snake-farm] Autodetect /usr/ccs/bin? In-Reply-To: <2m1y71nt9a.fsf@starship.python.net> References: <20021006115713.GB22400@lysator.liu.se> <200210061329.g96DTOs14718@pcp02138704pcs.reston01.va.comcast.net> <15778.6981.362515.852029@montanaro.dyndns.org> <2m1y71nt9a.fsf@starship.python.net> Message-ID: <15779.217.866575.83673@montanaro.dyndns.org> Michael> If this is your change: ... Michael> + add_dir_to_list(self.compiler.library_dirs, '/sw/lib') Michael> + add_dir_to_list(self.compiler.include_dirs, '/sw/include') yes, but prefaced with if sys.platform == "darwin": Michael> This is so easy, I don't think it's worth bothering with in the Michael> core... What's "the core" mean in this context? Do you mean that people should be expected to add this to setup.py themselves? Skip From oren-py-d@hishome.net Tue Oct 8 17:03:51 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Tue, 8 Oct 2002 12:03:51 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021008135549.GF5116@xs4all.nl> References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> <20021008135549.GF5116@xs4all.nl> Message-ID: <20021008160351.GA20990@hishome.net> On Tue, Oct 08, 2002 at 03:55:49PM +0200, Thomas Wouters wrote: > I'd also say rational-literals are not that important. Looking at my own > Python code, I very rarely need a floating-point literal to start with. > Strings, plenty, dicts and lists fairly often, integers every now and then, > but floating point numbers very rarely, and almost all of them are just > '0.0' or an integer expressed as float to force float-division. Most of my > float objects come from (library) functions that return them. Having nice-looking literals is important even if they are not actually typed in the source code too often. The literal form is also the repr() for all built-in numeric types so far. I don't think we should break that. What would your like to see as the repr() of a rational number? The answer to this will also determine what you type in your source. Note that repr(n) is not necessarily str(n): >>> repr(f) '0.59999999999999998' >>> str(f) '0.6' So we could have: >>> repr(r) '3/5r' # or 'rat(3, 6)' >>> str(r) '3/5' Oren From guido@python.org Tue Oct 8 17:09:57 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 12:09:57 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Tue, 08 Oct 2002 12:03:51 EDT." <20021008160351.GA20990@hishome.net> References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> <20021008135549.GF5116@xs4all.nl> <20021008160351.GA20990@hishome.net> Message-ID: <200210081609.g98G9v516159@odiug.zope.com> > So we could have: > > >>> repr(r) > '3/5r' # or 'rat(3, 6)' Surely you meant 'rat(3, 5)'. :-) > >>> str(r) > '3/5' I'd like at least one of those return '0.6' or '0.6r'. I think str(r) should return '0.6', and then repr(r) could return '3/5r'. For values that require approximation as decimal, I'd say use the same number of digits that str() of a float currently uses (about 12 I believe). So str(1/3r) should be '0.333333333333'. --Guido van Rossum (home page: http://www.python.org/~guido/) From jepler@unpythonic.net Tue Oct 8 17:47:28 2002 From: jepler@unpythonic.net (Jeff Epler) Date: Tue, 8 Oct 2002 11:47:28 -0500 Subject: [Python-Dev] read-only properties In-Reply-To: References: Message-ID: <20021008164723.GA22918@unpythonic.net> On Tue, Oct 08, 2002 at 11:37:46AM -0400, David Abrahams wrote: [...] > AttributeError: can't set attribute > > This struck me as a very poor error message, given that mechanism in > Boost.Python v1 used to say: [...] > AttributeError: 'l' attribute is read-only > > Is it worth doing something about this? The following comes close, in Python, but requires a setter object for each attribute: class SetROProperty: def __init__(self, name): self.name = name def __call__(self, obj, val): raise AttributeError, "%r attribute is read-only" % self.name class X(object): def get(self): return 'x' l = property(get, SetROProperty('l')) >>> x = X() >>> x.l 'x' >>> x.l = 1 Traceback (most recent call last): File "", line 1, in ? File "srop.py", line 5, in __call__ raise AttributeError, "%r attribute is read-only" % self.name AttributeError: 'l' attribute is read-only The key seems to be that property() doesn't know the name the property has. Jeff From richie@entrian.com Tue Oct 8 17:56:15 2002 From: richie@entrian.com (Richie Hindle) Date: Tue, 08 Oct 2002 17:56:15 +0100 Subject: [Python-Dev] Set-next-statement in Python debuggers In-Reply-To: <200210081542.g98FgIn15739@odiug.zope.com> References: <200210081542.g98FgIn15739@odiug.zope.com> Message-ID: > A backwards diff! He posted a backwards diff! Aargh! :-) Oops, mental note well and truly made... > if you add a C "getset" wrapper that > allows setting this only to safe values (i.e. within the same block > and pointing at the beginning of a bytecode) it might be deemed safe > enough. It should probably silently update f_lineno as well. OK, I'll have a look at doing that, and at creating a suitably convincing test suite. -- Richie Hindle richie@entrian.com From skip@pobox.com Tue Oct 8 18:18:55 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 8 Oct 2002 12:18:55 -0500 Subject: [Python-Dev] Warning about use of long double - what's the correct approach? In-Reply-To: <2m4rbxntfh.fsf@starship.python.net> References: <15778.647.849183.55911@montanaro.dyndns.org> <2m4rbxntfh.fsf@starship.python.net> Message-ID: <15779.4991.649997.288990@montanaro.dyndns.org> Michael> I thought that's what configure already did: Michael> case $ac_sys_system in Michael> Darwin*) Michael> OPT="$OPT -Wno-long-double -no-cpp-precomp";; Michael> esac Michael> If I've guessed wrong and you're not on OS X, give us some help :) You guessed right. In my situation, I generally configure with OPT=-O3 or OPT=-g, both of which prevent execution of this bit of code. Perhaps we should modify it so this is executed unconditionally. Skip From guido@python.org Tue Oct 8 18:27:35 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 13:27:35 -0400 Subject: [Python-Dev] Warning about use of long double - what's the correct approach? In-Reply-To: Your message of "Tue, 08 Oct 2002 12:18:55 CDT." <15779.4991.649997.288990@montanaro.dyndns.org> References: <15778.647.849183.55911@montanaro.dyndns.org> <2m4rbxntfh.fsf@starship.python.net> <15779.4991.649997.288990@montanaro.dyndns.org> Message-ID: <200210081727.g98HRZJ16526@odiug.zope.com> > Michael> I thought that's what configure already did: > > Michael> case $ac_sys_system in > Michael> Darwin*) > Michael> OPT="$OPT -Wno-long-double -no-cpp-precomp";; > Michael> esac > > Michael> If I've guessed wrong and you're not on OS X, give us some help :) > > You guessed right. In my situation, I generally configure with OPT=-O3 or > OPT=-g, both of which prevent execution of this bit of code. Perhaps we > should modify it so this is executed unconditionally. That's a longstanding nit: $OPT is used to specify some standard things as well as giving you a hook to override. I think we should introduce a separate variable. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Tue Oct 8 18:38:24 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 8 Oct 2002 12:38:24 -0500 Subject: [Python-Dev] Warning about use of long double - what's the correct approach? In-Reply-To: <200210081727.g98HRZJ16526@odiug.zope.com> References: <15778.647.849183.55911@montanaro.dyndns.org> <2m4rbxntfh.fsf@starship.python.net> <15779.4991.649997.288990@montanaro.dyndns.org> <200210081727.g98HRZJ16526@odiug.zope.com> Message-ID: <15779.6160.714341.875991@montanaro.dyndns.org> >>>>> "Guido" == Guido van Rossum writes: Guido> That's a longstanding nit: $OPT is used to specify some standard Guido> things as well as giving you a hook to override. I think we Guido> should introduce a separate variable. That makes good sense to me. For the purposes of backward compatibility, I suppose we shouldn't change OPT as the variable people set to tweak settings. I'll create a new BASEOPT variable and submit a patch. Skip From guido@python.org Tue Oct 8 18:51:30 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 13:51:30 -0400 Subject: [Python-Dev] Warning about use of long double - what's the correct approach? In-Reply-To: Your message of "Tue, 08 Oct 2002 12:38:24 CDT." <15779.6160.714341.875991@montanaro.dyndns.org> References: <15778.647.849183.55911@montanaro.dyndns.org> <2m4rbxntfh.fsf@starship.python.net> <15779.4991.649997.288990@montanaro.dyndns.org> <200210081727.g98HRZJ16526@odiug.zope.com> <15779.6160.714341.875991@montanaro.dyndns.org> Message-ID: <200210081751.g98HpUU16615@odiug.zope.com> > Guido> That's a longstanding nit: $OPT is used to specify some > Guido> standard things as well as giving you a hook to override. > Guido> I think we should introduce a separate variable. > > That makes good sense to me. For the purposes of backward > compatibility, I suppose we shouldn't change OPT as the variable > people set to tweak settings. I'll create a new BASEOPT variable > and submit a patch. OK! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Oct 8 19:25:59 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 14:25:59 -0400 Subject: [Python-Dev] read-only properties In-Reply-To: Your message of "Tue, 08 Oct 2002 11:47:28 CDT." <20021008164723.GA22918@unpythonic.net> References: <20021008164723.GA22918@unpythonic.net> Message-ID: <200210081826.g98IPx516833@odiug.zope.com> > The key seems to be that property() doesn't know the name the > property has. Oops, you're right. This makes it practically impossible to improve the error message (without the kind of trick that Jeff shows). --Guido van Rossum (home page: http://www.python.org/~guido/) From tim@zope.com Tue Oct 8 21:15:16 2002 From: tim@zope.com (Tim Peters) Date: Tue, 8 Oct 2002 16:15:16 -0400 Subject: [Python-Dev] tests failing in 2.2.2 In-Reply-To: Message-ID: [Guido, on the list(xrange(maxint // 4)) test] > That's strange -- that test is specifically designed to fail before it > ever gets to allocating memory. Can you track this down with a > debugger? [Martin] > Notice that this test does not strictly achieve that: This is true, although I'd say it plain doesn't achieve that, and leave "strictly" out of it. > with a four-byte pointer, you attempt to allocate 0x20000000L items (due > to roundupsize), which is small enough to try a realloc. In turn, it > tries to allocate 0x80000000L bytes, i.e. 2GB. On a 32-bit machine, it > is possible to allocate that much memory. Depending on platform, of course. Under MSVC6, it does call the platform realloc(), and it's the latter that returns a NULL pointer (the max you can alloc under Win32 is 0x7ffdefff). If we changed the test to use maxint // 2 instead, it would (just barely) trigger the _new_size <= ((~(size_t)0) / sizeof(type)) test in NRESIZE instead, and keep the platform realloc() out of it. Good enough? From guido@python.org Tue Oct 8 21:25:06 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 16:25:06 -0400 Subject: [Python-Dev] tests failing in 2.2.2 In-Reply-To: Your message of "Tue, 08 Oct 2002 16:15:16 EDT." References: Message-ID: <200210082025.g98KP6m24416@odiug.zope.com> > [Guido, on the list(xrange(maxint // 4)) test] > > That's strange -- that test is specifically designed to fail before it > > ever gets to allocating memory. Can you track this down with a > > debugger? > > [Martin] > > Notice that this test does not strictly achieve that: [Tim] > This is true, although I'd say it plain doesn't achieve that, and leave > "strictly" out of it. > > > with a four-byte pointer, you attempt to allocate 0x20000000L items (due > > to roundupsize), which is small enough to try a realloc. In turn, it > > tries to allocate 0x80000000L bytes, i.e. 2GB. On a 32-bit machine, it > > is possible to allocate that much memory. > > Depending on platform, of course. Under MSVC6, it does call the platform > realloc(), and it's the latter that returns a NULL pointer (the max you can > alloc under Win32 is 0x7ffdefff). > > If we changed the test to use maxint // 2 instead, it would (just barely) > trigger the > > _new_size <= ((~(size_t)0) / sizeof(type)) > > test in NRESIZE instead, and keep the platform realloc() out of it. > > Good enough? [Guido] I'd say so, except I fear that it might cause different problems (not that I know of any). I tried it on Linux and on Win98 and it works fine. I presume you've tested it on Win2K. But wasn't it specifically designed to catch a condition that was only true when the size was (approximately) maxint/4? --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Tue Oct 8 21:33:30 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 08 Oct 2002 16:33:30 -0400 Subject: [Python-Dev] tests failing in 2.2.2 Message-ID: <36d7e3d7e1.3d7e136d7e@icomcast.net> [Tim] >> If we changed the test to use maxint // 2 instead, it would >> (just barely) trigger the >> >> _new_size <= ((~(size_t)0) / sizeof(type)) >> >> test in NRESIZE instead, and keep the platform realloc() out of it. >> >> Good enough? [Guido] > I'd say so, except I fear that it might cause different problems > (not that I know of any). I tried it on Linux and on Win98 > and it works fine. I presume you've tested it on Win2K. Not really, but I watched what it *would* have done in the debugger. > But wasn't it specifically designed to catch a condition that was > only true when the size was (approximately) maxint/4? If that was the intent, then Martin was right that it's not catching what it intended to catch: as written, it goes to the platform realloc. The only place it *could* get caught before going to the platform realloc is in the snippet of code from NRESIZE I included above. The suggested change would (just barely) trigger that code instead, and we wouldn't invoked the platform realloc() at all then. There are no other interesting conditions I know of, or can find in the code. From Jack.Jansen@oratrix.com Tue Oct 8 21:37:41 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Tue, 8 Oct 2002 22:37:41 +0200 Subject: [Python-Dev] Python 2.2.2b1 checkin freeze In-Reply-To: <20021008135832.GA22587@epoch.metaslash.com> Message-ID: > On Mon, Oct 07, 2002 at 05:45:59PM -0400, Guido van Rossum wrote: >> I've committed the last changes for Python 2.2.2b1 -- the Misc/NEWS >> file. I'm now going to follow the rest of the PEP 102 script until >> the release is done. In the mean time, please no more checkins to the >> 2.2 branch! Sigh, just got around to giving 2.2.2b1 a first spin on MacOS9 (yes, I'm back since saturday but didn't have email access until yesterday, so it's only since this afternoon I'm aware of the 2.2.2 schedule), and it crashes the machine hard in the test suite after gobbling up all memory. The problem may be in test_class, or it could be the victim of an earlier memory problem, I'm still investigating. Ah, found it! It's the stack limit problem I ran into with 2.3a0 too. So, this particular fix should be okay to check in (it's in the mac-only .mcp project) but I'll wait for an okay from you anyway. If I find more problems I think the options are (I think) - I refrain from checking anything in until 2.2.2 for unix/win has been released, then I grab hold of the maint branch and check fixes in and we release MacPython 2.2.2 with a slightly different set of sources, or - You hold up the release process until I've found the problem and fixed it, and then we do 2.2.2b2 to make sure the fix doesn't break any other platforms. It's your call, -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From guido@python.org Tue Oct 8 21:48:48 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 16:48:48 -0400 Subject: [Python-Dev] Python 2.2.2b1 checkin freeze In-Reply-To: Your message of "Tue, 08 Oct 2002 22:37:41 +0200." References: Message-ID: <200210082048.g98Kmmh03773@odiug.zope.com> > > On Mon, Oct 07, 2002 at 05:45:59PM -0400, Guido van Rossum wrote: > >> I've committed the last changes for Python 2.2.2b1 -- the Misc/NEWS > >> file. I'm now going to follow the rest of the PEP 102 script until > >> the release is done. In the mean time, please no more checkins to the > >> 2.2 branch! Sorry, the freeze is over, since I did the beta release last night. I should've mentioned that. > Sigh, just got around to giving 2.2.2b1 a first spin on MacOS9 > (yes, I'm back since saturday but didn't have email access until > yesterday, so it's only since this afternoon I'm aware of the > 2.2.2 schedule), and it crashes the machine hard in the test > suite after gobbling up all memory. The problem may be in > test_class, or it could be the victim of an earlier memory > problem, I'm still investigating. Ah, found it! It's the stack > limit problem I ran into with 2.3a0 too. So, this particular fix > should be okay to check in (it's in the mac-only .mcp project) > but I'll wait for an okay from you anyway. Go ahead. You may also run into a problem in test_b1.py (allocating a list of size sys.maxint/4); Tim will fix in a change there that should fix it by making that sys.maxint/2 (!). > If I find more problems I think the options are (I think) > - I refrain from checking anything in until 2.2.2 for unix/win > has been released, then I grab hold of the maint branch and > check fixes in and we release MacPython 2.2.2 with a slightly > different set of sources, or > - You hold up the release process until I've found the problem > and fixed it, and then we do 2.2.2b2 to make sure the fix > doesn't break any other platforms. > > It's your call, Please do all the checkins on the Mac branch you want. For checkins in the rest of the code, I think you should use your normal caution but please fix things that are broken! --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Tue Oct 8 22:06:39 2002 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 8 Oct 2002 17:06:39 -0400 Subject: [Python-Dev] tests failing in 2.2.2 In-Reply-To: <200210082025.g98KP6m24416@odiug.zope.com> References: <200210082025.g98KP6m24416@odiug.zope.com> Message-ID: <20021008210639.GB22587@epoch.metaslash.com> On Tue, Oct 08, 2002 at 04:25:06PM -0400, Guido van Rossum wrote: > [Tim] > > > with a four-byte pointer, you attempt to allocate 0x20000000L items (due > > > to roundupsize), which is small enough to try a realloc. In turn, it > > > tries to allocate 0x80000000L bytes, i.e. 2GB. On a 32-bit machine, it > > > is possible to allocate that much memory. > > > > Depending on platform, of course. Under MSVC6, it does call the platform > > realloc(), and it's the latter that returns a NULL pointer (the max you can > > alloc under Win32 is 0x7ffdefff). > > > > If we changed the test to use maxint // 2 instead, it would (just barely) > > trigger the > > > > _new_size <= ((~(size_t)0) / sizeof(type)) > > > > test in NRESIZE instead, and keep the platform realloc() out of it. > > > > Good enough? > > [Guido] I'd say so, except I fear that it might cause different > problems (not that I know of any). I tried it on Linux and on Win98 > and it works fine. I presume you've tested it on Win2K. > > But wasn't it specifically designed to catch a condition that was only > true when the size was (approximately) maxint/4? The original problem was actually 1e9. The test is somewhat dubious, so anything to reduce the problems would be good. Perhaps the test should just be removed? Neal From tim.one@comcast.net Tue Oct 8 22:13:19 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 08 Oct 2002 17:13:19 -0400 Subject: [Python-Dev] tests failing in 2.2.2 Message-ID: <39c003b550.3b55039c00@icomcast.net> [Neal Norwitz] > The original problem was actually 1e9. Which is about maxint/2, which I checked in. > The test is somewhat dubious, so anything to reduce the problems > would be good. Perhaps the test should just be removed? So long as it's testing Python's overflow-checking code instead of the platform realloc(), it should be fine. It's probably still dubious on Win64 (where sys.maxint is still 2**31-1), but should be fine everywhere else). I'll worry about Win64 when somebody runs it <1/64 wink>. From neal@metaslash.com Tue Oct 8 22:16:33 2002 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 8 Oct 2002 17:16:33 -0400 Subject: [Python-Dev] tests failing in 2.2.2 In-Reply-To: <200210041500.g94F0XV25271@pcp02138704pcs.reston01.va.comcast.net> References: <20021004135948.GA30482@epoch.metaslash.com> <200210041500.g94F0XV25271@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021008211633.GA30812@epoch.metaslash.com> On Fri, Oct 04, 2002 at 11:00:33AM -0400, Guido van Rossum wrote: > [Neal Norwitz] > > Is anyone else having problems with 2.2.2 tests? > > > > minidom & sax fail for me on Linux: > > > > test_sax > > test test_sax skipped -- cannot import name quoteattr > > test_minidom > > Traceback (most recent call last): > > File "Lib/test/test_minidom.py", line 636, in ? > > func() > > File "Lib/test/test_minidom.py", line 208, in testAAA > > "setAttribute() sets ownerDocument") > > File "Lib/test/test_minidom.py", line 22, in confirm > > raise Exception > > Exception > > > > I'm off for a long weekend. I'll be back late Monday. > > Both tests work fine for me. I expect there's a problem with your > expat installation. Maybe Fred has a suggestion? I had an installed version of 2.2.2 from a while ago, in addition to the latest 2.2.2. I removed the installed version and all the tests pass now. Neal From neal@metaslash.com Tue Oct 8 22:27:15 2002 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 8 Oct 2002 17:27:15 -0400 Subject: [Python-Dev] segmentation fault with python2.3a0 from cvs In-Reply-To: References: <20021007173834.GA10088@codesourcery.com> Message-ID: <20021008212715.GB30812@epoch.metaslash.com> On Mon, Oct 07, 2002 at 08:03:56PM +0200, Gerson Kurz wrote: > Zack Weinberg wrote: > > > For what it's worth, this may be system-specific. I was able to > > import your test case under both 2.2.1 and 2.3a0 with no problem, on > > linux. The 2.3 compiler is a lot faster, which is nice. > > It seems like that. I compiled with DevStudio 6, and the import works (at > least in the debug build, but thats the only one I checked). I recompiled > python unter cygwin using "--with-pydebug", but to the same effect: cygwins' > python.exe raises a segmentation fault. So maybe, just maybe, its a fault of > cygwin ? > > $ uname -a > CYGWIN_NT-5.0 DREIZEHN 1.3.12(0.54/3/2) 2002-07-06 02:16 i686 unknown This may be relevant, from Lib/test/test_b1.py: # Note: This test is expected to SEGV under Cygwin 1.3.12 or # earlier due to a newlib bug. See the following mailing list # thread for the details: # http://sources.redhat.com/ml/newlib/2002/msg00369.html In Cygwin, malloc() was treating the size as signed which caused a crash. Perhaps you are experiencing something similar? Have you tried with the latest Cygwin? Neal From Jack.Jansen@oratrix.com Tue Oct 8 22:46:59 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Tue, 8 Oct 2002 23:46:59 +0200 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 Message-ID: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> Aside from the usual suspects I have three tests failing in MacPython 2.2.2: test_atexit, test_email and test_httplib. The first one should be easy to fix: backport the revision 1.7 fix. Would that cause any problems on other platforms? The other two I have no idea of what could be causing them, so I need suggestions. I can't copy/paste, so I'll retype a bit. test_email: assertion failed, 2042 != 2. from line 1793 in test_crlf_separation. Could this have something to do with MacPython 2.2's "poor man's universal newlines"? (MacPython 2.2.X will read both \r and \n as \n for files opened in text mode, but not \r\n. This quick hack was put in to make 2.2 usable on OSX). test_httplib: mismatch in lines 9-12 of expected output. It wanted two header: Set-Cookie: lines, but got nothing. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From skip@pobox.com Tue Oct 8 22:54:40 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 8 Oct 2002 16:54:40 -0500 Subject: [Python-Dev] Where is this flag coming from? Message-ID: <15779.21536.368403.957238@montanaro.dyndns.org> On my MacOS 10.2 system an unexpected flag is added to the link line for the bsddb module: gcc -bundle -bundle_loader python.exe \ build/temp.darwin-6.1-Power\ Macintosh-2.3/bsddbmodule.o \ -L/sw/lib -L/sw/lib -L/usr/local/lib -Wl,-R/sw/lib -ldb-4.1 \ -o build/lib.darwin-6.1-Power\ Macintosh-2.3/bsddb.so None of the other extension modules are linked with "-Wl,-R/sw/lib". I don't find "-R" in either setup.py or configure. Any idea where this is coming from and how to get rid of it? Skip From barry@python.org Tue Oct 8 23:25:26 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 8 Oct 2002 18:25:26 -0400 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 References: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> Message-ID: <15779.23382.921583.645228@gargle.gargle.HOWL> >>>>> "JJ" == Jack Jansen writes: JJ> test_email: assertion failed, 2042 != 2. from line 1793 in JJ> test_crlf_separation. Could this have something to do with JJ> MacPython 2.2's "poor man's universal newlines"? (MacPython JJ> 2.2.X will read both \r and \n as \n for files opened in text JJ> mode, but not \r\n. This quick hack was put in to make 2.2 JJ> usable on OSX). Undoubtably. Guido and I had an inkling about that this morning but didn't follow up. I suppose I should fire up MacOSX and see what I can work out, but if you have suggestions I'm all ears. -Barry From barry@python.org Tue Oct 8 23:30:25 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 8 Oct 2002 18:30:25 -0400 Subject: [Python-Dev] Where is this flag coming from? References: <15779.21536.368403.957238@montanaro.dyndns.org> Message-ID: <15779.23681.462147.252222@gargle.gargle.HOWL> >>>>> "SM" == Skip Montanaro writes: SM> On my MacOS 10.2 system an unexpected flag is added to the SM> link line for the bsddb module: | gcc -bundle -bundle_loader python.exe \ | build/temp.darwin-6.1-Power\ Macintosh-2.3/bsddbmodule.o \ | -L/sw/lib -L/sw/lib -L/usr/local/lib -Wl,-R/sw/lib -ldb-4.1 \ | -o build/lib.darwin-6.1-Power\ Macintosh-2.3/bsddb.so SM> None of the other extension modules are linked with SM> "-Wl,-R/sw/lib". I don't find "-R" in either setup.py or SM> configure. Any idea where this is coming from and how to get SM> rid of it? Don't you remember, Skip!? I think we worked this out when we were looking at the linking problems with bsddb, since the from-source Berkeley install doesn't put its libs in a normally searched location. These flags get added because of the runtime_library_dirs arg in the Extension constructor for bsddbmodule.c. Are you saying that the flags are wrong, the paths are wrong, the flags are unnecessary, or something else? -Barry From jrw@pobox.com Tue Oct 8 23:34:05 2002 From: jrw@pobox.com (John Williams) Date: Tue, 08 Oct 2002 17:34:05 -0500 Subject: [Python-Dev] read-only properties References: <20021008164723.GA22918@unpythonic.net> <200210081826.g98IPx516833@odiug.zope.com> Message-ID: <3DA35D5D.9010503@pobox.com> Guido van Rossum wrote: >>The key seems to be that property() doesn't know the name the >>property has. > > > Oops, you're right. This makes it practically impossible to improve > the error message (without the kind of trick that Jeff shows). > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev Why not have the property find its name in the class's __dict__, something like this: (in class property) def __set__(self, obj, value): if self.fset is not None: self.fset(obj, value) else: for key, value in type(obj).__dict__.iteritems(): if value is self: raise AttributeError, "Property '" + key + "' is read-only." # No name? Strange put possible I guess. raise AttributeError, "can't set property" From greg@cosc.canterbury.ac.nz Wed Oct 9 00:36:23 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Wed, 09 Oct 2002 12:36:23 +1300 (NZDT) Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021008160351.GA20990@hishome.net> Message-ID: <200210082336.g98NaNp18660@oma.cosc.canterbury.ac.nz> Oren Tirosh : > What would your like to see as the repr() of a rational number? The answer > to this will also determine what you type in your source. I think it's the other way around -- what you type in the source determines what repr() should return. If a constructor, e.g. rat(2,3) is used, then the repr() should be likewise. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From barry@python.org Wed Oct 9 00:50:12 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 8 Oct 2002 19:50:12 -0400 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 References: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> Message-ID: <15779.28468.831725.1313@gargle.gargle.HOWL> >>>>> "JJ" == Jack Jansen writes: JJ> Aside from the usual suspects I have three tests failing in JJ> MacPython 2.2.2: test_atexit, test_email and test_httplib. Oops, looks like I'm also getting a segfault in test_re. -Barry From oren-py-d@hishome.net Wed Oct 9 01:02:12 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Wed, 9 Oct 2002 02:02:12 +0200 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <200210081609.g98G9v516159@odiug.zope.com>; from guido@python.org on Tue, Oct 08, 2002 at 12:09:57PM -0400 References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> <20021008135549.GF5116@xs4all.nl> <20021008160351.GA20990@hishome.net> <200210081609.g98G9v516159@odiug.zope.com> Message-ID: <20021009020212.A31472@hishome.net> On Tue, Oct 08, 2002 at 12:09:57PM -0400, Guido van Rossum wrote: > > > '3/5' > > I'd like at least one of those return '0.6' or '0.6r'. Why should any of them go through a potetially lossy transformation? If you need to approximate a ratio as a finite decimal fraction it should be explicit: str(float(r)). I think it will also make a good visual cue to always use rational notation for rationals and decimal fractions for floats. The result of repr() should eval() back to exactly the same object. The result of str() should be the 'pretty' representation because this is the form displayed by print statements and %s formatting. I find '2/3' prettier than '0.666666666667' pretty-is-in-the-eyes-of-the-beholder-ly yours, Oren From barry@python.org Wed Oct 9 01:04:31 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 8 Oct 2002 20:04:31 -0400 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 References: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> <15779.28468.831725.1313@gargle.gargle.HOWL> Message-ID: <15779.29327.201832.109788@gargle.gargle.HOWL> >>>>> "BAW" == Barry A Warsaw writes: BAW> Oops, looks like I'm also getting a segfault in test_re. Let's see if cut-n-paste works. This is under gdb on OSX 10.2 with a simple "configure; make". Jack, you're not seeing this are you? -Barry @freewill[~/Desktop/src/python22:217]% gdb python.exe GNU gdb 5.1-20020408 (Apple version gdb-231) (Tue Aug 13 21:37:39 GMT 2002) Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "powerpc-apple-macos10". Reading symbols for shared libraries .... done (gdb) run Lib/test/test_re.py Starting program: /Users/barry/Desktop/src/python22/python.exe Lib/test/test_re.py [Switching to process 5135 thread 0xb03] Reading symbols for shared libraries ............. done Reading symbols for shared libraries . done Running tests on re.search and re.match Running tests on re.sub Running tests on symbolic references Running tests on re.subn Running tests on re.split Running tests on re.findall Running tests on re.match Running tests on re.escape Pickling a RegexObject instance Reading symbols for shared libraries . done Reading symbols for shared libraries . done Test engine limitations Program received signal EXC_BAD_ACCESS, Could not access memory. 0x0009cfb8 in mark_save (state=0xbffff1c0, lo=1, hi=1) at Modules/_sre.c:290 290 { (gdb) where #0 0x0009cfb8 in mark_save (state=0xbffff1c0, lo=1, hi=1) at Modules/_sre.c:290 #1 0x0009c168 in sre_match (state=0xbffff240, pattern=0x3ea47a, level=4068) at Modules/_sre.c:1070 [... lots of frames in sre_match ... ] #4067 0x0009c0b8 in sre_match (state=0xbffff240, pattern=0x3ea466, level=1) at Modules/_sre.c:1026 #4068 0x0009e928 in pattern_match (self=0x3ea440, args=0x0, kw=0x1) at Modules/_sre.c:1694 #4069 0x000591c0 in PyObject_Call (func=0xbffff240, arg=0x0, kw=0x1) at Objects/abstract.c:1688 #4070 0x00074c4c in do_call (func=0x3ea050, pp_stack=0xbffff6b4, na=1, nk=1528) at Python/ceval.c:3271 #4071 0x000725e0 in eval_frame (f=0x3ba3b0) at Python/ceval.c:2009 #4072 0x00073a50 in PyEval_EvalCodeEx (co=0x386cb0, globals=0x0, locals=0x1, args=0x3ba50c, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=1, closure=0x0) at Python/ceval.c:2595 #4073 0x00074b30 in fast_function (func=0xbffff240, pp_stack=0x54, n=1, na=4104272, nk=1) at Python/ceval.c:3170 #4074 0x000726fc in eval_frame (f=0x3ba3b0) at Python/ceval.c:2034 #4075 0x00073a50 in PyEval_EvalCodeEx (co=0x386cb0, globals=0x0, locals=0x1, args=0x3ba50c, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2595 #4076 0x00076574 in PyEval_EvalCode (co=0xbffff240, globals=0x0, locals=0x1) at Python/ceval.c:481 #4077 0x0002a650 in run_node (n=0x3ba50c, filename=0x0, globals=0x39bab2, locals=0x1, flags=0x9bad0) at Python/pythonrun.c:1079 #4078 0x00029dfc in PyRun_SimpleFileExFlags (fp=0xa0006950, filename=0xbffffcd5 "Lib/test/test_re.py", closeit=1, flags=0x386cb0) at Python/pythonrun.c:685 #4079 0x00006344 in Py_Main (argc=-2013265336, argv=0xbffffbe8) at Modules/main.c:364 #4080 0x00002254 in _start (argc=2, argv=0xbffffbe8, envp=0xbffffbf4) at /SourceCache/Csu/Csu-45/crt.c:267 #4081 0x000020d4 in start () Yes, that stack trace is that deep. -Barry From barry@python.org Wed Oct 9 01:11:33 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 8 Oct 2002 20:11:33 -0400 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 References: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> Message-ID: <15779.29749.886457.639973@gargle.gargle.HOWL> >>>>> "JJ" == Jack Jansen writes: JJ> Aside from the usual suspects I have three tests failing in JJ> MacPython 2.2.2: test_atexit, test_email and test_httplib. JJ> test_email: assertion failed, 2042 != 2. from line 1793 in JJ> test_crlf_separation. Could this have something to do with JJ> MacPython 2.2's "poor man's universal newlines"? (MacPython JJ> 2.2.X will read both \r and \n as \n for files opened in text JJ> mode, but not \r\n. This quick hack was put in to make 2.2 JJ> usable on OSX). Re-reading this, are you saying that this failure is on MacOS9 only? That test passes just fine on OSX 10.2 and the data looks just fine to me, so this must be an OS9 failure only. Maybe the thing to do is to open msg_26.txt in binary mode, but only on MacOS9? Maybe we should add a sys.platform test to test_crlf_separation()? -Barry From guido@python.org Wed Oct 9 01:34:29 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 20:34:29 -0400 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 In-Reply-To: Your message of "Tue, 08 Oct 2002 23:46:59 +0200." <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> References: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> Message-ID: <200210090034.g990YTx26342@pcp02138704pcs.reston01.va.comcast.net> > Aside from the usual suspects I have three tests failing in > MacPython 2.2.2: test_atexit, test_email and test_httplib. > > The first one should be easy to fix: backport the revision 1.7 > fix. Would that cause any problems on other platforms? No problemo. > The other two I have no idea of what could be causing them, so I > need suggestions. I can't copy/paste, so I'll retype a bit. > > test_email: assertion failed, 2042 != 2. from line 1793 in > test_crlf_separation. > Could this have something to do with MacPython 2.2's "poor man's > universal newlines"? (MacPython 2.2.X will read both \r and \n > as \n for files opened in text mode, but not \r\n. This quick > hack was put in to make 2.2 usable on OSX). Is in Barry's capable hands. > test_httplib: mismatch in lines 9-12 of expected output. It > wanted two header: Set-Cookie: lines, but got nothing. Probably a line ending problem. I notice that those two lines end with \r\n! I don't understand why the test uses \r\n. I propose to change all the \r\n to \n in test_httplib.py and ditto in output/test_httplib. If you can get that working, feel free to check it in; or I can do it. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 9 01:38:48 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 08 Oct 2002 20:38:48 -0400 Subject: [Python-Dev] read-only properties In-Reply-To: Your message of "Tue, 08 Oct 2002 17:34:05 CDT." <3DA35D5D.9010503@pobox.com> References: <20021008164723.GA22918@unpythonic.net> <200210081826.g98IPx516833@odiug.zope.com> <3DA35D5D.9010503@pobox.com> Message-ID: <200210090038.g990cmK26359@pcp02138704pcs.reston01.va.comcast.net> > Why not have the property find its name in the class's __dict__, > something like this: > > (in class property) > def __set__(self, obj, value): > if self.fset is not None: > self.fset(obj, value) > else: > for key, value in type(obj).__dict__.iteritems(): > if value is self: > raise AttributeError, "Property '" + key + "' is read-only." > # No name? Strange put possible I guess. > raise AttributeError, "can't set property" You'd have to search base classes too, in MRO order. Doable, but makes AttributeError very expensive. (This can be a concern if it is in fact being caught.) It could also be misleading -- the same property object may occur more than once, and you don't know which one was being set. I think it's best to rely on the traceback. If the traceback looks like Traceback (most recent call last): File "foo.py", line 42, in bar self.foo = 42 AttributeError: can't set attribute the problem should be clear enough. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Wed Oct 9 03:18:46 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 08 Oct 2002 22:18:46 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: <20021009020212.A31472@hishome.net> Message-ID: [Guido, on 3/5r] > I'd like at least one of those return '0.6' or '0.6r'. [Oren Tirosh] > Why should any of them go through a potetially lossy transformation? 0.6r is exact, although it's hard to know whether Guido was hoping to preserve that or not. > If you need to approximate a ratio as a finite decimal fraction it should > be explicit: str(float(r)). > I think it will also make a good visual cue to always use > rational notation for rationals and decimal fractions for floats. The trailing 'r' *is* rational notation, and stuff like this probably isn't a good idea for str(): > #e6.02e-45 301/50000000000000000000000000000000000000000000000 > 6.02e-45r would be clearer to virtually anyone. > (+ 312 (/ 1 3)) 937/3 > would likely be clearer to virtually anyone as 312+1/3r too. But most of all, I agree that if you're working with rationals, you don't want to lose information silently, not even in a "pleasant" string. > The result of repr() should eval() back to exactly the same object. The > result of str() should be the 'pretty' representation because this is the > form displayed by print statements and %s formatting. I find > '2/3' prettier than '0.666666666667' 2/3r works for both for me. 0.666...7 doesn't work for me at all as a stringification of a rational (if I want to approximate, I'll ask for an approximation). From skip@pobox.com Wed Oct 9 04:24:36 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 8 Oct 2002 22:24:36 -0500 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <15779.23681.462147.252222@gargle.gargle.HOWL> References: <15779.21536.368403.957238@montanaro.dyndns.org> <15779.23681.462147.252222@gargle.gargle.HOWL> Message-ID: <15779.41332.251210.360418@montanaro.dyndns.org> SM> None of the other extension modules are linked with "-Wl,-R/sw/lib". SM> I don't find "-R" in either setup.py or configure. Any idea where SM> this is coming from and how to get rid of it? BAW> Don't you remember, Skip!? I think we worked this out when we were BAW> looking at the linking problems with bsddb, since the from-source BAW> Berkeley install doesn't put its libs in a normally searched BAW> location. I remember having problems with bsddb, which I think were fixed. -Wl,-R/sw/lib is clearly wrong on MacOS X though. Deleting it and running the link command manually works. BAW> Are you saying that the flags are wrong, the paths are wrong, the BAW> flags are unnecessary, or something else? The flags are incorrect: ... building 'bsddb' extension gcc -bundle -bundle_loader python.exe \ build/temp.darwin-6.1-Power\ Macintosh-2.3/bsddbmodule.o \ -L/sw/lib -L/home/skip/local/lib -L/sw/lib -L/usr/local/lib \ -Wl,-R/sw/lib -ldb-4.1 \ -o build/lib.darwin-6.1-Power\ Macintosh-2.3/bsddb.so ld: unknown flag: -R/sw/lib ... Skip From tim@multitalents.net Wed Oct 9 05:54:44 2002 From: tim@multitalents.net (Tim Rice) Date: Tue, 8 Oct 2002 21:54:44 -0700 (PDT) Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <15779.41332.251210.360418@montanaro.dyndns.org> Message-ID: On Tue, 8 Oct 2002, Skip Montanaro wrote: > > SM> None of the other extension modules are linked with "-Wl,-R/sw/lib". > SM> I don't find "-R" in either setup.py or configure. Any idea where > SM> this is coming from and how to get rid of it? > > BAW> Don't you remember, Skip!? I think we worked this out when we were > BAW> looking at the linking problems with bsddb, since the from-source > BAW> Berkeley install doesn't put its libs in a normally searched > BAW> location. > > I remember having problems with bsddb, which I think were fixed. > -Wl,-R/sw/lib is clearly wrong on MacOS X though. Deleting it and running > the link command manually works. > > BAW> Are you saying that the flags are wrong, the paths are wrong, the > BAW> flags are unnecessary, or something else? > The -R flag in not supported on all systems. 2.2.2b1 build is broken on SCO Open Server now too. > The flags are incorrect: > > ... > building 'bsddb' extension > gcc -bundle -bundle_loader python.exe \ > build/temp.darwin-6.1-Power\ Macintosh-2.3/bsddbmodule.o \ > -L/sw/lib -L/home/skip/local/lib -L/sw/lib -L/usr/local/lib \ > -Wl,-R/sw/lib -ldb-4.1 \ > -o build/lib.darwin-6.1-Power\ Macintosh-2.3/bsddb.so > ld: unknown flag: -R/sw/lib > ... > > Skip > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > -- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net From barry@python.org Wed Oct 9 05:58:18 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 9 Oct 2002 00:58:18 -0400 Subject: [Python-Dev] Where is this flag coming from? References: <15779.41332.251210.360418@montanaro.dyndns.org> Message-ID: <15779.46954.672730.965163@gargle.gargle.HOWL> >>>>> "TR" == Tim Rice writes: TR> The -R flag in not supported on all systems. TR> 2.2.2b1 build is broken on SCO Open Server now too. Shouldn't distutils be taught what to do with runtime_library_dirs for SCO and MacOSX? -Barry From oren-py-d@hishome.net Wed Oct 9 09:13:06 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Wed, 9 Oct 2002 10:13:06 +0200 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: ; from tim.one@comcast.net on Tue, Oct 08, 2002 at 10:18:46PM -0400 References: <20021009020212.A31472@hishome.net> Message-ID: <20021009101306.A4307@hishome.net> On Tue, Oct 08, 2002 at 10:18:46PM -0400, Tim Peters wrote: > > If you need to approximate a ratio as a finite decimal fraction it should > > be explicit: str(float(r)). > > I think it will also make a good visual cue to always use > > rational notation for rationals and decimal fractions for floats. > > The trailing 'r' *is* rational notation, and stuff like this probably isn't > a good idea for str(): I'm not sure I follow. What 'stuff' is not a good idea for str() and why? By rational notation I meant the division form, not the trailing r or any other syntax used by the language to indicate the type. > > > #e6.02e-45 > 301/50000000000000000000000000000000000000000000000 > > > > 6.02e-45r would be clearer to virtually anyone. Again, I'm not sure I follow your logic. How likely is an arbitrary rational number to have a denominator ending with lots of zeros? > > (+ 312 (/ 1 3)) > 937/3 > > > > would likely be clearer to virtually anyone as 312+1/3r too. Yes, in this case it is much clearer. > But most of all, I agree that if you're working with rationals, you don't > want to lose information silently, not even in a "pleasant" string. > > > The result of repr() should eval() back to exactly the same object. The > > result of str() should be the 'pretty' representation because this is the > > form displayed by print statements and %s formatting. I find > > '2/3' prettier than '0.666666666667' > > 2/3r works for both for me. 0.666...7 doesn't work for me at all as a > stringification of a rational (if I want to approximate, I'll ask for an > approximation). Works for repr, str, or both? The str() of long omits the trailing L. I don't think rationals should show the trailing r. Oren From mwh@python.net Wed Oct 9 10:34:27 2002 From: mwh@python.net (Michael Hudson) Date: 09 Oct 2002 10:34:27 +0100 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 In-Reply-To: barry@python.org's message of "Tue, 8 Oct 2002 19:50:12 -0400" References: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> <15779.28468.831725.1313@gargle.gargle.HOWL> Message-ID: <2mfzvgnefw.fsf@starship.python.net> barry@python.org (Barry A. Warsaw) writes: > >>>>> "JJ" == Jack Jansen writes: > > JJ> Aside from the usual suspects I have three tests failing in > JJ> MacPython 2.2.2: test_atexit, test_email and test_httplib. > > Oops, looks like I'm also getting a segfault in test_re. This is normal; run "ulimit -s 2048" before running the test suite. Cheers, M. -- please realize that the Common Lisp community is more than 40 years old. collectively, the community has already been where every clueless newbie will be going for the next three years. so relax, please. -- Erik Naggum, comp.lang.lisp From stephen@xemacs.org Wed Oct 9 13:10:36 2002 From: stephen@xemacs.org (Stephen J. Turnbull) Date: Wed, 09 Oct 2002 21:10:36 +0900 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: <200210071220.g97CKRG06496@pcp02138704pcs.reston01.va.comcast.net> (Guido van Rossum's message of "Mon, 07 Oct 2002 08:20:27 -0400") References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <200210071220.g97CKRG06496@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <8765wbak3n.fsf@tleepslib.sk.tsukuba.ac.jp> >>>>> "Guido" =3D=3D Guido van Rossum writes: >> Did you try to open the file in IDLE? What other tools are you >> using? Guido> XEmacs. The particular version or configuration I'm using Guido> apparently doesn't recogize the coding cookie. No current XEmacs recognizes cookies in the second line. I tried it, it broke, I backed it out, had a bad attack of day job, and forgot to fix the patch. :-( This may[1] work in 21.4.10. The code is moderately complicated because we have to deal with all the different linebreaking conventions in the raw buffer. I wish I could work on an editor that only needs to deal with Lisp, Python, and XML.... You will need to use Mule-UCS (the package that provides un-define) to get Unicode (any flavor of UTF) support in XEmacs 21.4. You probably don't want latin-unity if you're playing with cookies, because it currently incorrectly complains about cookies that are OK. (This is also EOL breakage. :-=FE) Footnotes:=20 [1] Assuming the new patch isn't broken in a new way, and has to be backed out. --=20 Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac= .jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JA= PAN My nostalgia for Icon makes me forget about any of the bad things. I don't have much nostalgia for Perl, so its faults I remember. Scott Gilbert c.l.= py From stephen@xemacs.org Wed Oct 9 13:19:32 2002 From: stephen@xemacs.org (Stephen J. Turnbull) Date: Wed, 09 Oct 2002 21:19:32 +0900 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: <200210071220.g97CKRG06496@pcp02138704pcs.reston01.va.comcast.net> (Guido van Rossum's message of "Mon, 07 Oct 2002 08:20:27 -0400") References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <200210071220.g97CKRG06496@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <873crfajor.fsf@tleepslib.sk.tsukuba.ac.jp> Oops. I wrote: >> No current XEmacs recognizes cookies in the second line. and >> You will need to use Mule-UCS [for UTF-8] I suspect this means that there will be glitches in cookie support for UTF-8, even in l.1. I'll keep lurking to see what people are saying on py-dev, or if you really really want support in XEmacs, write me directly with bug reports and example files and I'll see what I can do (I still hate cookies but love Python more than ever!) [No, Barry, I dunno nothing about RFC 2047 + UTF-8, that's been on my list for a long time, too.] -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN My nostalgia for Icon makes me forget about any of the bad things. I don't have much nostalgia for Perl, so its faults I remember. Scott Gilbert c.l.py From arigo@tunes.org Wed Oct 9 13:57:26 2002 From: arigo@tunes.org (Armin Rigo) Date: Wed, 9 Oct 2002 14:57:26 +0200 (CEST) Subject: [Python-Dev] Set-next-statement in Python debuggers In-Reply-To: Message-ID: Hello Richie, On Tue, 8 Oct 2002 richie@entrian.com wrote: > (...) A nasty consequence is that you can write Python code that causes > Python to seg-fault, but you have to be doing some fairly advanced stuff > for that to happen. You can already crash the interpreter with pure Python code, for example via the new.code() constructor or by writing crappy .pyc files. On Linux you can also open("/proc/self/mem", "w"). I don't think that people get their hands on frame objects by pure chance, but the possibility exists. Moreover, conditionally allowing changes to f_lasti is limiting and complex because of the stack and block stack. For safety I'd consider writing the frame-object-modifying code in a C extension module, carefully documented as "don't use this". Armin From neal@metaslash.com Wed Oct 9 14:11:42 2002 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 9 Oct 2002 09:11:42 -0400 Subject: [Python-Dev] Re: [snake-farm] Re: [farm-report] Daily Snake Farm report In-Reply-To: <49c9447952.4795249c94@icomcast.net> References: <49c9447952.4795249c94@icomcast.net> Message-ID: <20021009131142.GG30812@epoch.metaslash.com> On Wed, Oct 09, 2002 at 03:59:29AM -0400, Tim Peters wrote: > [Anders Qvist] > > ... > > check_all("ntpath") > > File "../../python/dist/src/Lib/test/test___all__.py", line 15, > > in check_all > > exec "import %s" % modname in names > > File "", line 1, in ? > > File "/home/quest/tmp/compile- > > python/python/dist/src/Lib/ntpath.py", line 482, in ? > > supports_unicode_filenames = sys.getwindowsversion()[3] >= 2 > > AttributeError: 'module' object has no attribute 'getwindowsversion' > > Grrr. I expect I just fixed this, but can't test it now on Linux for > you. The person who checked this change in probably didn't realize > that ntpath was imported on any system other than Windows. I'm > surprised the bug went uncaught for so long! (Ya, it was less than a > day, but still ...) :-) I've been running 2.2.2 only. I just tried to run 2.3 and had some problems. __all__ is ok, though. The first problem is in regrtest. This exception causes regrtest to crash. test_pyexpat Traceback (most recent call last): File "./Lib/test/regrtest.py", line 877, in ? main() File "./Lib/test/regrtest.py", line 229, in main ok = runtest(test, generate, verbose, quiet, testdir) File "./Lib/test/regrtest.py", line 392, in runtest output = cfp.getvalue() File "/home/neal/build/python/dist/src/Lib/StringIO.py", line 195, in getvalue self.buf += ''.join(self.buflist) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 36: ordinal not in range(128) I put a try/except around line 392 in regrtest and then had 2 failures: test_pyexpat had no output (the unicode problem I guess) and test_rotor: test test_rotor produced unexpected output: ********************************************************************** *** mismatch between lines 2-3 of expected output and lines 2-3 of actual output: - '\xb5q\x90\xa3\xe8IT\xd1\x07\xda\x1d\xdc\xdf' + 'qqqqq\x90 ... Some unicode problem complaining about these 2 chars: \xcb and \xe2 which are printed as e umlaut and e circumflex. Maybe this is just me again though? Neal From mwh@python.net Wed Oct 9 14:20:50 2002 From: mwh@python.net (Michael Hudson) Date: 09 Oct 2002 14:20:50 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib ntpath.py,1.50,1.51 References: Message-ID: <2md6qjd9zh.fsf@starship.python.net> tim_one@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Lib > In directory usw-pr-cvs1:/tmp/cvs-serv28728/python/Lib > > Modified Files: > ntpath.py > Log Message: > Don't try to access sys.getwindowsversion unless it exists (ntpath is > imported on systems other than Windows, and in particular is imported > by test___all__; the compile farm reported that all Linux tests failed > due to this; isn't anyone in PythonDevLand running CVS on Linux?!). Yes, but I do sleep occasionally... Cheers, M. -- it's not that perl programmers are idiots, it's that the language rewards idiotic behavior in a way that no other language or tool has ever done -- Erik Naggum, comp.lang.lisp From skip@pobox.com Wed Oct 9 14:54:49 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 9 Oct 2002 08:54:49 -0500 Subject: [Python-Dev] test_locale failing on MacOSX w/ 2.2.2 Message-ID: <15780.13609.477242.561460@montanaro.dyndns.org> Just got this test failure this morning on MacOS 10.2 running w/ 2.2.2 from CVS (or very nearly so): % ./python.exe ../Lib/test/regrtest.py test_locale test_locale test test_locale produced unexpected output: ********************************************************************** *** lines 2-4 of actual output doesn't appear in expected output after line 1: + '%f' % 1024 == '1024.000000' != '1,024.000000' + '%+10.f' % -4200 == ' -4200' != ' -4,200' + '%-10.f' % 4200 == '4200 ' != '4,200 ' ********************************************************************** Skip From list-python@ccraig.org Wed Oct 9 15:16:46 2002 From: list-python@ccraig.org (Christopher A. Craig) Date: 09 Oct 2002 10:16:46 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: References: Message-ID: Tim Peters writes: > 2/3r works for both for me. 0.666...7 doesn't work for me at all as > a stringification of a rational (if I want to approximate, I'll ask > for an approximation). I agree. I initially thought about returning a decimal with str(), but for that reason chose not to. It might be nice to have a method that returned the best floating point decimal approximation to within n decimal digits or shorter, but I don't want a builtin to make an arbitrary decision on where to cut off precision. I do like the idea of str() returning the form 5+3/2r, though. -- Christopher A. Craig Python is an excellent language for learning object orientation. (It also happens to be my favorite OO scripting language.) Sriram Srinivasan -- "Advanced Perl Programming" From guido@python.org Wed Oct 9 15:52:43 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 09 Oct 2002 10:52:43 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Wed, 09 Oct 2002 02:02:12 +0200." <20021009020212.A31472@hishome.net> References: <200210080018.g980IA706850@oma.cosc.canterbury.ac.nz> <200210081241.g98CfVA23876@mercur.uphs.upenn.edu> <20021008135549.GF5116@xs4all.nl> <20021008160351.GA20990@hishome.net> <200210081609.g98G9v516159@odiug.zope.com> <20021009020212.A31472@hishome.net> Message-ID: <200210091452.g99EqiR10950@odiug.zope.com> > > > '3/5' > > > > I'd like at least one of those return '0.6' or '0.6r'. > > Why should any of them go through a potetially lossy transformation? Human factors. Same reason why printing a float rounds to about 12 digits. The "accuracy" of 123456789/234567890 is lost on the eye of the beholder -- I'd have to count digits to tell whether that's smaller or larger than one! > If you need to approximate a ratio as a finite decimal fraction it > should be explicit: str(float(r)). Nah, if you want to show the numerator and denominator you should use repr(r). > I think it will also make a good visual cue to always use rational > notation for rationals and decimal fractions for floats. What do you mean by that? What's wrong with using 0.6r for a rational number? > The result of repr() should eval() back to exactly the same > object. The result of str() should be the 'pretty' representation > because this is the form displayed by print statements and %s > formatting. I find '2/3' prettier than '0.666666666667' But there are very few rationals where that's really true. Most of them look more like 1753/811. --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Wed Oct 9 15:49:44 2002 From: dave@boost-consulting.com (David Abrahams) Date: 09 Oct 2002 10:49:44 -0400 Subject: [Python-Dev] Issue with 2.2.2b1 and MinGW-2.0 Message-ID: So I thought I'd try testing the latests Boost.Python with 2.2.2 and ALL of the compilers I have available to me this morning, and it turned up this interesting data point: What I get from the Windows installer differs from what I get when I build Python using MS Visual Studio 6 in a way that affects whether MinGW-2.0 can link extension modules with it out-of-the-box. The one that I get from the installer fails to link (errors below) unless I make the magic incantations specified here: http://www.python.org/doc/current/inst/non-ms-compilers.html#SECTION000312000000000000000 The one that I build myself works just fine "out-of-the-box". --- These were the link errors I got from MinGW-2.0 g++ using the Python I got from the Windows installer: Creating library file: c:\build\libs\python\build\bin\boost_python.dll\mingw-2.0\debug\runtime-link-dynamic\boost_python.lib c:\build\libs\python\build\bin\boost_python.dll\mingw-2.0\debug\runtime-link-dynamic\numeric.obj: In function `ZN5boost6python7numeric53_GLOBAL__N_.._.._.._libs_python_src_numeric.cpp2dT8hb18throw_load_failureEv': c:/boost/libs/python/test/../../../libs/python/src/numeric.cpp:29: undefined reference to `_imp__PyExc_ImportError' c:\build\libs\python\build\bin\boost_python.dll\mingw-2.0\debug\runtime-link-dynamic\numeric.obj: In function `ZN5boost6python7numeric53_GLOBAL__N_.._.._.._libs_python_src_numeric.cpp2dT8hb4loadEb': c:/boost/libs/python/test/../../../libs/python/src/numeric.cpp:57: undefined reference to `_imp__PyType_Type' c:/boost/libs/python/test/../../../libs/python/src/numeric.cpp:57: undefined reference to `_imp__PyType_Type' c:\build\libs\python\build\bin\boost_python.dll\mingw-2.0\debug\runtime-link-dynamic\list.obj: In function `ZN5boost6python6detail9list_base4callERKNS0_3api6objectE': c:/boost/libs/python/test/../../../libs/python/src/list.cpp:12: undefined reference to `_imp__PyList_Type' c:\build\libs\python\build\bin\boost_python.dll\mingw-2.0\debug\runtime-link-dynamic\list.obj: In function `ZN5boost6python6detail9list_base6appendERKNS0_3api6objectE': c:/boost/libs/python/test/../../../libs/python/src/list.cpp:29: undefined reference to `_imp__PyList_Type' c:\build\libs\python\build\bin\boost_python.dll\mingw-2.0\debug\runtime-link-dynamic\list.obj: In function `ZN5boost6python6detail9list_base6insertEiRKNS0_3api6objectE': c:/boost/libs/python/test/../../../libs/python/src/list.cpp:65: undefined reference to `_imp__PyList_Type' c:\build\libs\python\build\bin\boost_python.dll\mingw-2.0\debug\runtime-link-dynamic\list.obj: In function `ZN5boost6python6detail9list_base7reverseEv': c:/boost/libs/python/test/../../../libs/python/src/list.cpp:106: undefined reference to `_imp__PyList_Type' c:\build\libs\python\build\bin\boost_python.dll\mingw-2.0\debug\runtime-link-dynamic\list.obj: In function `ZN5boost6python6detail9list_base4sortEv': c:/boost/libs/python/test/../../../libs/python/src/list.cpp:119: undefined reference to `_imp__PyList_Type' c:\build\libs\python\build\bin\boost_python.dll\mingw-2.0\debug\runtime-link-dynamic\long.obj: In function `ZN5boost6python6detail9long_base4callERKNS0_3api6objectE': c:/boost/libs/python/test/../../../libs/python/src/long.cpp:12: undefined reference to `_imp__PyLong_Type' c:\build\libs\python\build\bin\boost_python.dll\mingw-2.0\debug\runtime-link-dynamic\long.obj: In function `ZN5boost6python6detail9long_base4callERKNS0_3api6objectES6_': -- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From guido@python.org Wed Oct 9 16:36:31 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 09 Oct 2002 11:36:31 -0400 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: Your message of "Wed, 09 Oct 2002 00:58:18 EDT." <15779.46954.672730.965163@gargle.gargle.HOWL> References: <15779.41332.251210.360418@montanaro.dyndns.org> <15779.46954.672730.965163@gargle.gargle.HOWL> Message-ID: <200210091536.g99FaVa11272@odiug.zope.com> > TR> The -R flag in not supported on all systems. > TR> 2.2.2b1 build is broken on SCO Open Server now too. > > Shouldn't distutils be taught what to do with runtime_library_dirs for > SCO and MacOSX? My thoughts exactly. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 9 16:55:13 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 09 Oct 2002 11:55:13 -0400 Subject: [Python-Dev] Set-next-statement in Python debuggers In-Reply-To: Your message of "Wed, 09 Oct 2002 14:57:26 +0200." References: Message-ID: <200210091555.g99FtD011873@odiug.zope.com> > On Tue, 8 Oct 2002 richie@entrian.com wrote: > > (...) A nasty consequence is that you can write Python code that causes > > Python to seg-fault, but you have to be doing some fairly advanced stuff > > for that to happen. [Armin] > You can already crash the interpreter with pure Python code, for example > via the new.code() constructor or by writing crappy .pyc files. Yes, and I occasionally lose sleep over those. I definitely don't want to add more loopholes like that, and I'd like to fix those. In the past, the new module was optional for precisely this reason (and so is the dl module). I would like to have a check on .pyc files (or on unmarshalled code objects, really). > On Linux you can also open("/proc/self/mem", "w"). That doesn't mean that python should abandon the policy "a core dump is always considered Python's fault unless proven otherwise." Protecting you against using *external* tools may not be possible; but Python should not include features (like writable code object attributes) that can cause crashes when used inexpertly. > I don't think that people get their hands on frame objects by pure > chance, but the possibility exists. Moreover, conditionally > allowing changes to f_lasti is limiting and complex because of the > stack and block stack. For safety I'd consider writing the > frame-object-modifying code in a C extension module, carefully > documented as "don't use this". That's a reasonable solution: f_lasti should be read-only from Python code, but you can write an extension that can write it. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 9 17:13:09 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 09 Oct 2002 12:13:09 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Your message of "Wed, 09 Oct 2002 10:16:46 EDT." References: Message-ID: <200210091613.g99GD9812324@odiug.zope.com> > Tim Peters writes: > > 2/3r works for both for me. 0.666...7 doesn't work for me at all as > > a stringification of a rational (if I want to approximate, I'll ask > > for an approximation). > > I agree. I initially thought about returning a decimal with str(), > but for that reason chose not to. It might be nice to have a method > that returned the best floating point decimal approximation to within > n decimal digits or shorter, but I don't want a builtin to make an > arbitrary decision on where to cut off precision. > > I do like the idea of str() returning the form 5+3/2r, though. Well, that would completely kills any possibility of ever making 1/3 return a rational (currently the main argument against it is the expectation amongst the current user base that it will be a float). I really don't think that this would be reasonable at all: Python 3.0 (#2345, Mar 12 2005, 11:32:10) Type "help", "copyright", "credits" or "license" for more information. >>> print 25/7 25/7 >>> --Guido van Rossum (home page: http://www.python.org/~guido/) From tim@multitalents.net Wed Oct 9 17:15:15 2002 From: tim@multitalents.net (Tim Rice) Date: Wed, 9 Oct 2002 09:15:15 -0700 (PDT) Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <15779.46954.672730.965163@gargle.gargle.HOWL> Message-ID: On Wed, 9 Oct 2002, Barry A. Warsaw wrote: > > >>>>> "TR" == Tim Rice writes: > > TR> The -R flag in not supported on all systems. > TR> 2.2.2b1 build is broken on SCO Open Server now too. > > Shouldn't distutils be taught what to do with runtime_library_dirs for > SCO and MacOSX? It gets tricky trying to automagicly use the -R flag. As an example, the default install dir for OpenSSL is /usr/local/ssl. So on platforms that support -R you would think you could use -L/usr/local/ssl/lib -R/usr/local/ssl/lib -lssl -lcrypto OpenUNIX supports -R but if you did not build shared libraries for OpenSSL, the -R/usr/local/ssl/lib will break the build. Then there is the whole issue of what non system dirs to include. Maybe you have OpenSSL 0.9.6g in /usr/local/ssl but you're testing 0.9.7 that's installed in some other dir. If setup.py automagicly adds ssl to build the '_socket' extension, how does it know which version to use? How would it even find the one not in /usr/local/ssl? Normally you would configure --with-ssl-dir=/some_path_to_ssl As one who has done ports to many Open Source projects, I found python one of the more difficult ones because the build process uses python. So for a new platform (or broken build) you have to know python to build python. > > -Barry > -- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net From tim@zope.com Wed Oct 9 17:42:27 2002 From: tim@zope.com (Tim Peters) Date: Wed, 9 Oct 2002 12:42:27 -0400 Subject: [Python-Dev] Snapshot win32all builds of interest? In-Reply-To: Message-ID: [Mark Hammond] > OK - first such snapshot done. See the link from > starship.python.net/crew/mhammond > > This version installs no shortcuts, does not compile .pyc files etc - you > are pretty much on your own. Pythonwin\start_pythonwin.pyw is > installed to start Pythonwin, but you must do so manually. > > Please let me know if you try it. I tried it. I'd say something useful, except as far as I could tell, it worked . Thanks! Have you talked w/ the PBF about getting builds tested? FYI, the current PLabs Python uninstaller arranges to delete .pyc and .pyo files via these WISE lines in the installer: Add "File Tree: %MAINDIR%\*.pyc" to INSTALL.LOG Add "File Tree: %MAINDIR%\*.pyo" to INSTALL.LOG These arrange to (recursively) delete all .pyc and .pyo files at uninstall time, and "do the right thing": any directories left empty at the end are magically removed. From martin@v.loewis.de Wed Oct 9 18:02:32 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 09 Oct 2002 19:02:32 +0200 Subject: [Python-Dev] test_locale failing on MacOSX w/ 2.2.2 In-Reply-To: <15780.13609.477242.561460@montanaro.dyndns.org> References: <15780.13609.477242.561460@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > ********************************************************************** > *** lines 2-4 of actual output doesn't appear in expected output after line 1: > + '%f' % 1024 == '1024.000000' != '1,024.000000' > + '%+10.f' % -4200 == ' -4200' != ' -4,200' > + '%-10.f' % 4200 == '4200 ' != '4,200 ' > ********************************************************************** That's most likely an OS X bug, for not properly supporting the en_US locale. Various systems fail this test, we have always declared those problems 3rdparty. Regards, Martin From thomas.heller@ion-tof.com Wed Oct 9 18:05:03 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed, 9 Oct 2002 19:05:03 +0200 Subject: [Python-Dev] Snapshot win32all builds of interest? References: Message-ID: <03ba01c26fb6$02668780$e000a8c0@thomasnotebook> From: "Tim Peters" > [Mark Hammond] > > OK - first such snapshot done. See the link from > > starship.python.net/crew/mhammond > > > > This version installs no shortcuts, does not compile .pyc files etc - you > > are pretty much on your own. Pythonwin\start_pythonwin.pyw is > > installed to start Pythonwin, but you must do so manually. > > > > Please let me know if you try it. > > I tried it. I'd say something useful, except as far as I could tell, it > worked . Thanks! Have you talked w/ the PBF about getting builds > tested? Same for me (except that I see a lot of these FutureWarnings about hex constants > sys.maxint, but you already knew this). Thomas From martin@v.loewis.de Wed Oct 9 18:05:04 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 09 Oct 2002 19:05:04 +0200 Subject: [Python-Dev] Issue with 2.2.2b1 and MinGW-2.0 In-Reply-To: References: Message-ID: David Abrahams writes: > So I thought I'd try testing the latests Boost.Python with 2.2.2 and > ALL of the compilers I have available to me this morning, and it > turned up this interesting data point: David, Was this for information only, or did you want readers to infer some conclusion? Regards, Martin From martin@v.loewis.de Wed Oct 9 18:10:54 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 09 Oct 2002 19:10:54 +0200 Subject: [Python-Dev] Snapshot win32all builds of interest? In-Reply-To: References: Message-ID: "Tim Peters" writes: > I tried it. I'd say something useful, except as far as I could tell, it > worked . Thanks! I've had a few problems. Running it in a console gives the messages Failed to connect to Pythonwin|System. Error 0 mh\Pythonwin\pywin\scintilla\view.py:18: DeprecationWarning: the regex module is deprecated; please use the re module import regex E:\Pyinst\python\lib\regsub.py:15: DeprecationWarning: the regsub module is deprecated; please use re.sub() DeprecationWarning) which appear to be harmless; I'll report them anyway in case you are unaware of them. More troubling is that Find (Ctrl-F) does not work: I get Traceback (most recent call last): File "mh\Pythonwin\pywin\scintilla\view.py", line 349, in OnCmdEditFind ImportError: No module named find Any suggestions? Regards, Martin From dave@boost-consulting.com Wed Oct 9 18:21:55 2002 From: dave@boost-consulting.com (David Abrahams) Date: 09 Oct 2002 13:21:55 -0400 Subject: [Python-Dev] Issue with 2.2.2b1 and MinGW-2.0 In-Reply-To: References: Message-ID: martin@v.loewis.de (Martin v. Loewis) writes: > David Abrahams writes: > > > So I thought I'd try testing the latests Boost.Python with 2.2.2 and > > ALL of the compilers I have available to me this morning, and it > > turned up this interesting data point: > > David, > > Was this for information only, or did you want readers to infer some > conclusion? It's mostly informational. I don't think it's worth holding up the 2.2.2 release over, but somebody else might think it was especially worth investigating. -- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From guido@python.org Wed Oct 9 19:17:51 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 09 Oct 2002 14:17:51 -0400 Subject: [Python-Dev] Re: [snake-farm] Re: [farm-report] Daily Snake Farm report In-Reply-To: Your message of "Wed, 09 Oct 2002 09:11:42 EDT." <20021009131142.GG30812@epoch.metaslash.com> References: <49c9447952.4795249c94@icomcast.net> <20021009131142.GG30812@epoch.metaslash.com> Message-ID: <200210091817.g99IHqW16465@odiug.zope.com> > The first problem is in regrtest. This exception causes regrtest to crash. > > test_pyexpat > Traceback (most recent call last): > File "./Lib/test/regrtest.py", line 877, in ? > main() > File "./Lib/test/regrtest.py", line 229, in main > ok = runtest(test, generate, verbose, quiet, testdir) > File "./Lib/test/regrtest.py", line 392, in runtest > output = cfp.getvalue() > File "/home/neal/build/python/dist/src/Lib/StringIO.py", line 195, in getvalue > self.buf += ''.join(self.buflist) > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 36: ordinal not in range(128) > > I put a try/except around line 392 in regrtest and then had 2 failures: > > test_pyexpat had no output (the unicode problem I guess) and test_rotor: > > test test_rotor produced unexpected output: > ********************************************************************** > *** mismatch between lines 2-3 of expected output and lines 2-3 of actual output: > - '\xb5q\x90\xa3\xe8IT\xd1\x07\xda\x1d\xdc\xdf' > + 'qqqqq\x90 > ... > > Some unicode problem complaining about these 2 chars: \xcb and \xe2 > which are printed as e umlaut and e circumflex. > > Maybe this is just me again though? I saw this too. The fix was to add a setlocale() call to test___all__.py. Read my checkin comments and weep. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Wed Oct 9 19:18:40 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 9 Oct 2002 14:18:40 -0400 Subject: [Python-Dev] Where is this flag coming from? References: <15779.46954.672730.965163@gargle.gargle.HOWL> Message-ID: <15780.29440.641827.692637@gargle.gargle.HOWL> >>>>> "TR" == Tim Rice writes: TR> It gets tricky trying to automagicly use the -R flag. As an TR> example, the default install dir for OpenSSL is TR> /usr/local/ssl. So on platforms that support -R you would TR> think you could use -L/usr/local/ssl/lib -R/usr/local/ssl/lib TR> -lssl -lcrypto OpenUNIX supports -R but if you did not build TR> shared libraries for OpenSSL, the -R/usr/local/ssl/lib will TR> break the build. Since there seems to be so little agreement among *nixes about how to do this, it seems like the only/best place to codify all this knowledge is in distutils. I still think specifying runtime_library_dirs to the Extension constructor is the right thing to do. But maybe on some systems distutils ought to ignore it, or at least check to make sure that the dynamic libraries it wants to link against are actually present. TR> Then there is the whole issue of what non system dirs to TR> include. Maybe you have OpenSSL 0.9.6g in /usr/local/ssl but TR> you're testing 0.9.7 that's installed in some other dir. If TR> setup.py automagicly adds ssl to build the '_socket' TR> extension, how does it know which version to use? How would it TR> even find the one not in /usr/local/ssl? Normally you would TR> configure --with-ssl-dir=/some_path_to_ssl I would hope that setup.py would match the library version with the header version, but I'm sure it's not easy or foolproof. Look at the machinations setup.py goes through to build bsddbmodule. I still think it's best to do this so that Python has a chance of building out of the box for the widest possible audience. It's impossible to get 100% right. TR> As one who has done ports to many Open Source projects, I TR> found python one of the more difficult ones because the build TR> process uses python. So for a new platform (or broken build) TR> you have to know python to build python. Python's also a pretty complicated piece of software, what with each extension module linking against non-standard libraries presenting its own set of complications. But what tool that you can portably count on being present would you use instead? -Barry From quest@lysator.liu.se Wed Oct 9 19:34:33 2002 From: quest@lysator.liu.se (Anders Qvist) Date: Wed, 9 Oct 2002 20:34:33 +0200 Subject: [Python-Dev] Re: [snake-farm] Re: [farm-report] Daily Snake Farm report In-Reply-To: <200210091817.g99IHqW16465@odiug.zope.com> References: <49c9447952.4795249c94@icomcast.net> <20021009131142.GG30812@epoch.metaslash.com> <200210091817.g99IHqW16465@odiug.zope.com> Message-ID: <20021009183432.GQ22400@lysator.liu.se> On Wed, Oct 09, 2002 at 02:17:51PM -0400, Guido van Rossum wrote: > I saw this too. The fix was to add a setlocale() call to > test___all__.py. Read my checkin comments and weep. The intel/linux-boxen in the green again: http://www.lysator.liu.se/~sfarmer/xeno/latest.html -- Anders "Quest" Qvist "We've all heard that a million monkeys banging on a million typewriters will eventually reproduce the entire works of Shakespeare. Now, thanks to the Internet, we know this is not true." -- Robert Wilensky From guido@python.org Wed Oct 9 19:50:54 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 09 Oct 2002 14:50:54 -0400 Subject: [Python-Dev] Multibyte repr() In-Reply-To: Your message of "Wed, 09 Oct 2002 14:17:51 EDT." <200210091817.g99IHqW16465@odiug.zope.com> References: <49c9447952.4795249c94@icomcast.net> <20021009131142.GG30812@epoch.metaslash.com> <200210091817.g99IHqW16465@odiug.zope.com> Message-ID: <200210091850.g99Iosb17345@odiug.zope.com> > I saw this too. The fix was to add a setlocale() call to > test___all__.py. Read my checkin comments and weep. I told this all to Tim, and he had one comment. The repr() function of an 8-bit string can now return characters with the high bit set. This was the direct cause of the failures. It was introduced in the following patch: ---------------------------- revision 2.190 date: 2002/10/07 13:55:50; author: loewis; state: Exp; lines: +68 -15 Patch #479898: Use multibyte C library for printing strings if available. ---------------------------- Was this really a good idea??? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 9 20:04:08 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 09 Oct 2002 15:04:08 -0400 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: Your message of "Wed, 09 Oct 2002 14:50:54 EDT." Message-ID: <200210091904.g99J48O17461@odiug.zope.com> > I told this all to Tim, and he had one comment. The repr() function > of an 8-bit string can now return characters with the high bit set. > This was the direct cause of the failures. It was introduced in the > following patch: > > ---------------------------- > revision 2.190 > date: 2002/10/07 13:55:50; author: loewis; state: Exp; lines: +68 -15 > Patch #479898: Use multibyte C library for printing strings if available. > ---------------------------- > > Was this really a good idea??? Here's an example of what I mean. Python 2.2: >>> u = u'\u1f40' >>> s = u.encode('utf8') >>> s '\xe1\xbd\x80' >>> Python 2.3: >>> u = u'\u1f40' >>> s = u.encode('utf8') >>> s 'á½\x80' >>> The latter output is not helpful, because the encoding of s is not the locale's encoding. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Wed Oct 9 20:14:20 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 9 Oct 2002 15:14:20 -0400 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 References: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> <15779.28468.831725.1313@gargle.gargle.HOWL> <2mfzvgnefw.fsf@starship.python.net> Message-ID: <15780.32780.635036.756155@gargle.gargle.HOWL> >>>>> "MH" == Michael Hudson writes: >> "JJ" == Jack Jansen writes: >> JJ> Aside from the usual suspects I have three tests failing in >> JJ> MacPython 2.2.2: test_atexit, test_email and test_httplib. >> Oops, looks like I'm also getting a segfault in test_re. MH> This is normal; run "ulimit -s 2048" before running the test MH> suite. Thanks, that works. Would it make sense to add this to the test suite some how, either by adding a ulimit call to the makefile, or adding some resource.setrlimit() call to site.py or test_{,s}re.py? -Barry From fredrik@pythonware.com Wed Oct 9 20:19:21 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 9 Oct 2002 21:19:21 +0200 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 References: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com><15779.28468.831725.1313@gargle.gargle.HOWL><2mfzvgnefw.fsf@starship.python.net> <15780.32780.635036.756155@gargle.gargle.HOWL> Message-ID: <095701c26fc8$c8bf5580$ced241d5@hagrid> barry wrote: > MH> This is normal; run "ulimit -s 2048" before running the test > MH> suite. > > Thanks, that works. Would it make sense to add this to the test suite > some how, either by adding a ulimit call to the makefile, or adding > some resource.setrlimit() call to site.py or test_{,s}re.py? or s?re.py... From fredrik@pythonware.com Wed Oct 9 20:21:34 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 9 Oct 2002 21:21:34 +0200 Subject: [Python-Dev] Multibyte repr() References: <49c9447952.4795249c94@icomcast.net> <20021009131142.GG30812@epoch.metaslash.com> <200210091817.g99IHqW16465@odiug.zope.com> <200210091850.g99Iosb17345@odiug.zope.com> Message-ID: <095a01c26fc9$162a8470$ced241d5@hagrid> guido wrote: > Was this really a good idea??? first impression: it's a really bad idea. From martin@v.loewis.de Wed Oct 9 20:58:27 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 09 Oct 2002 21:58:27 +0200 Subject: [Python-Dev] Multibyte repr() In-Reply-To: <200210091850.g99Iosb17345@odiug.zope.com> References: <49c9447952.4795249c94@icomcast.net> <20021009131142.GG30812@epoch.metaslash.com> <200210091817.g99IHqW16465@odiug.zope.com> <200210091850.g99Iosb17345@odiug.zope.com> Message-ID: Guido van Rossum writes: > revision 2.190 > date: 2002/10/07 13:55:50; author: loewis; state: Exp; lines: +68 -15 > Patch #479898: Use multibyte C library for printing strings if available. > ---------------------------- > > Was this really a good idea??? I think the submitter wanted to get "proper" output in interactive mode for strings. For most cases, it might be sufficient to active the code for tp_print only, not for tp_repr. Would that be better? Regards, Martin From guido@python.org Wed Oct 9 21:13:43 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 09 Oct 2002 16:13:43 -0400 Subject: [Python-Dev] Multibyte repr() In-Reply-To: Your message of "Wed, 09 Oct 2002 21:58:27 +0200." References: <49c9447952.4795249c94@icomcast.net> <20021009131142.GG30812@epoch.metaslash.com> <200210091817.g99IHqW16465@odiug.zope.com> <200210091850.g99Iosb17345@odiug.zope.com> Message-ID: <200210092013.g99KDhe26705@odiug.zope.com> > > revision 2.190 > > date: 2002/10/07 13:55:50; author: loewis; state: Exp; lines: +68 -15 > > Patch #479898: Use multibyte C library for printing strings if available. > > ---------------------------- > > > > Was this really a good idea??? > > I think the submitter wanted to get "proper" output in interactive > mode for strings. For most cases, it might be sufficient to active the > code for tp_print only, not for tp_repr. > > Would that be better? Not really, because then it would break the equivalence between repr() and entering a value at the command line. It would also cause surprises because sometimes repr() is invoked anyway (when there's a container that doesn't have a tp_print function). I wouldn't mind this so much if GNU readline didn't do an evil thing by calling setlocale(). Maybe the readline C extension should restore the locale? This patch seems to take care of it, except I'm not sure if we can assume that and setlocale() are always available. This doesn't seem to have any ill effects on the operation of GNU readline: ////////////////////////////////////////////////////////////////////// *** readline.c 2 Aug 2002 02:27:13 -0000 2.52 --- readline.c 9 Oct 2002 20:10:33 -0000 *************** *** 11,16 **** --- 11,17 ---- #include #include #include + #include /* GNU readline definitions */ #undef HAVE_CONFIG_H /* Else readline/chardefs.h includes strings.h */ *************** *** 538,543 **** --- 539,545 ---- static void setup_readline(void) { + char *saved_locale = NULL; using_history(); rl_readline_name = "python"; *************** *** 570,576 **** --- 572,580 ---- * XXX: A bug in the readline-2.2 library causes a memory leak * inside this function. Nothing we can do about it. */ + saved_locale = setlocale(LC_CTYPE, NULL); rl_initialize(); + setlocale(LC_CTYPE, saved_locale); /* Restore locale */ } ////////////////////////////////////////////////////////////////////// --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Wed Oct 9 21:34:06 2002 From: tim.one@comcast.net (Tim Peters) Date: Wed, 09 Oct 2002 16:34:06 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: Message-ID: [Christopher A. Craig] > I agree. I initially thought about returning a decimal with str(), > but for that reason chose not to. I expect you'll have to in the end, though. Getting an explicit ratio of multi-thousand digit integers really does suck for a default ... > It might be nice to have a method that returned the best floating point > decimal approximation to within n decimal digits or shorter, but I don't > want a builtin to make an arbitrary decision on where to cut off > precision. If you haven't yet, you should strive to understand Moshe Zadka's prototype rational implementation in Python CVS nondist/sandbox/ (which has all the approximation functions you could need). Unfortunately, the SourceForge Python Numerics list: https://sourceforge.net/mailarchive/forum.php?forum_id=2280 appears to have lost almost all the voluminous discussions that went into that, retaining mostly a random sampling of spam. From martin@v.loewis.de Wed Oct 9 21:55:08 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 09 Oct 2002 22:55:08 +0200 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <15780.29440.641827.692637@gargle.gargle.HOWL> References: <15779.46954.672730.965163@gargle.gargle.HOWL> <15780.29440.641827.692637@gargle.gargle.HOWL> Message-ID: barry@python.org (Barry A. Warsaw) writes: > Since there seems to be so little agreement among *nixes about how to > do this, it seems like the only/best place to codify all this > knowledge is in distutils. I still think specifying > runtime_library_dirs to the Extension constructor is the right thing > to do. But maybe on some systems distutils ought to ignore it, or at > least check to make sure that the dynamic libraries it wants to link > against are actually present. I think both is difficult: It is hard to find out what option to use on what system, and how to find out whether a shared library is present. As a starting point, you have to decide whether you want a conservative strategy (white list of systems you know how to steer), or an optimistic strategy (black list of systems on which the standard strategy fails). For 2.2.2, backing out any setup.py changes to add runtime_library_dirs should be taking into consideration. Regards, Martin From martin@v.loewis.de Wed Oct 9 22:00:55 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 09 Oct 2002 23:00:55 +0200 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: <200210091904.g99J48O17461@odiug.zope.com> References: <200210091904.g99J48O17461@odiug.zope.com> Message-ID: Guido van Rossum writes: > >>> u = u'\u1f40' > >>> s = u.encode('utf8') > >>> s > 'a=\x80' > >>> > > The latter output is not helpful, because the encoding of s is not the > locale's encoding. [Somehow, the accents got lost in your message] It isn't helpful, but it isn't strictly wrong, either. In this specific case, people are used to see utf-8 being interpreted as Latin-1 - that form of "mojibake" is very common, so they will know what happened. I question whether the hex representation is more helpful: it depends on how you need to interpret the result you get. Regards, Martin From martin@v.loewis.de Wed Oct 9 22:07:23 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 09 Oct 2002 23:07:23 +0200 Subject: [Python-Dev] Multibyte repr() In-Reply-To: <200210092013.g99KDhe26705@odiug.zope.com> References: <49c9447952.4795249c94@icomcast.net> <20021009131142.GG30812@epoch.metaslash.com> <200210091817.g99IHqW16465@odiug.zope.com> <200210091850.g99Iosb17345@odiug.zope.com> <200210092013.g99KDhe26705@odiug.zope.com> Message-ID: Guido van Rossum writes: > This patch seems to take care of it, except I'm not sure if we can > assume that and setlocale() are always available. This > doesn't seem to have any ill effects on the operation of GNU readline: The patch looks fine, but you need to wrap it with HAVE_LOCALE_H and HAVE_SETLOCALE; both are already tested in configure.in. Regards, Martin From guido@python.org Wed Oct 9 22:07:33 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 09 Oct 2002 17:07:33 -0400 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: Your message of "Wed, 09 Oct 2002 23:00:55 +0200." References: <200210091904.g99J48O17461@odiug.zope.com> Message-ID: <200210092107.g99L7XA26891@odiug.zope.com> > Guido van Rossum writes: > > > >>> u = u'\u1f40' > > >>> s = u.encode('utf8') > > >>> s > > 'a=\x80' > > >>> > > > > The latter output is not helpful, because the encoding of s is not the > > locale's encoding. [Martin] > [Somehow, the accents got lost in your message] > > It isn't helpful, but it isn't strictly wrong, either. In this > specific case, people are used to see utf-8 being interpreted as > Latin-1 - that form of "mojibake" is very common, so they will know > what happened. > > I question whether the hex representation is more helpful: it depends > on how you need to interpret the result you get. Well, if you *want* to see the hex codes for all non-ASCII characters, repr() used to be your friend. No more. If you *want* to see the printable characters, you could always use print. I'd be okay with this change if the default locale wasn't changed by readline. Did you see my patch for that? Then people who want to see their encoding from repr() can learn to put import locale locale.setlocale(locale.LC_CTYPE, "") in their $PYTHONSTARTUP file (or in their app's main()). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 9 22:18:12 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 09 Oct 2002 17:18:12 -0400 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: Your message of "Wed, 09 Oct 2002 22:55:08 +0200." References: <15779.46954.672730.965163@gargle.gargle.HOWL> <15780.29440.641827.692637@gargle.gargle.HOWL> Message-ID: <200210092118.g99LICj26939@odiug.zope.com> [Martin] > For 2.2.2, backing out any setup.py changes to add > runtime_library_dirs should be taking into consideration. This sounds like a conservative solution. There is exactly one such option, for ssl. The checkin message: ---------------------------- revision 1.73.4.9 date: 2002/09/30 14:42:29; author: bwarsaw; state: Exp; lines: +2 -1 Add "runtime_library_dirs = ssl_libs" to the _socket Extension specification so that the proper runtime ld.so path gets compiled into the extension. This fixes _socket for Solaris systems with libssl and libcrypto in non-standard locations and should be fine for other systems as well. Closes SF bug #565710. Forward port candidate for Python 2.3 (I'll work on that). ---------------------------- The diff (apart from a change to __version__) is: *************** *** 364,369 **** --- 364,370 ---- exts.append( Extension('_socket', ['socketmodule.c'], include_dirs = ssl_incs, library_dirs = ssl_libs, + runtime_library_dirs = ssl_libs, libraries = ['ssl', 'crypto'], define_macros = [('USE_SSL',1)] ) ) else: Maybe we should be more conservative and add this option only for Solaris? How does one detect Solaris in setup.py? How about this patch (on top of the above one, not replacing it)? *** setup.py 7 Oct 2002 10:38:33 -0000 1.73.4.10 --- setup.py 9 Oct 2002 21:15:04 -0000 *************** *** 361,370 **** if (ssl_incs is not None and ssl_libs is not None): exts.append( Extension('_socket', ['socketmodule.c'], include_dirs = ssl_incs, library_dirs = ssl_libs, ! runtime_library_dirs = ssl_libs, libraries = ['ssl', 'crypto'], define_macros = [('USE_SSL',1)] ) ) else: --- 361,373 ---- if (ssl_incs is not None and ssl_libs is not None): + rtlibs = None + if platform.startswith('sunos'): + rtlibs = ssl_libs exts.append( Extension('_socket', ['socketmodule.c'], include_dirs = ssl_incs, library_dirs = ssl_libs, ! runtime_library_dirs = rtlibs, libraries = ['ssl', 'crypto'], define_macros = [('USE_SSL',1)] ) ) else: I hope that Skip/Barry can test this on MacOS X and Martin on Solaris! Note: the 2.3 setup.py contains other uses of runtime_library_dirs, which probably need to be treated the same way. Skip, can you cook up patches there? (The 2.2.2 fix for ssl also needs to be forward-ported!) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 9 22:28:05 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 09 Oct 2002 17:28:05 -0400 Subject: [Python-Dev] Multibyte repr() In-Reply-To: Your message of "Wed, 09 Oct 2002 23:07:23 +0200." References: <49c9447952.4795249c94@icomcast.net> <20021009131142.GG30812@epoch.metaslash.com> <200210091817.g99IHqW16465@odiug.zope.com> <200210091850.g99Iosb17345@odiug.zope.com> <200210092013.g99KDhe26705@odiug.zope.com> Message-ID: <200210092128.g99LS6W27043@odiug.zope.com> > > This patch seems to take care of it, except I'm not sure if we can > > assume that and setlocale() are always available. This > > doesn't seem to have any ill effects on the operation of GNU readline: > > The patch looks fine, but you need to wrap it with HAVE_LOCALE_H and > HAVE_SETLOCALE; both are already tested in configure.in. OK, done and checked in. --Guido van Rossum (home page: http://www.python.org/~guido/) From nhodgson@bigpond.net.au Wed Oct 9 23:04:04 2002 From: nhodgson@bigpond.net.au (Neil Hodgson) Date: Thu, 10 Oct 2002 08:04:04 +1000 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net><200210071220.g97CKRG06496@pcp02138704pcs.reston01.va.comcast.net> <8765wbak3n.fsf@tleepslib.sk.tsukuba.ac.jp> Message-ID: <005001c26fdf$c8609c40$3da48490@neil> Stephen J. Turnbull: > No current XEmacs recognizes cookies in the second line. I tried it, > it broke, I backed it out, had a bad attack of day job, and forgot to > fix the patch. :-( Where did the coding cookie originally come from? It seems to have arrived in Python from somewhere else, probably Emacs but I couldn't find a reference in PEP-263. Googling with a couple of sets of relevant terms didn't reveal anything good within the top references. I have just implemented the start of this feature in my SciTE editor and would like to ensure it is compatible with other editors. Neil From mats@laplaza.org Thu Oct 10 00:48:15 2002 From: mats@laplaza.org (Mats Wichmann) Date: Wed, 09 Oct 2002 17:48:15 -0600 Subject: [Python-Dev] 2.2.2b fails test_minidom on Itanium Linux In-Reply-To: <20021009022002.15248.70185.Mailman@mail.python.org> Message-ID: <5.1.0.14.1.20021009174456.01fce830@204.151.72.2> Been tied up since the beta was announced; the 2.2.2 CVS of this morning, which I assume is very close if not identical to the beta, fails with "lines 2-217 of output missing" on the minidom test on Itanium (ia64) Linux. I presume this is due to the test dying or some resource missing so it doesn't get started. No chance to check deeper just now. This was fine just a bit ago when I sent Guido some feedback pre-beta. Mats From barry@python.org Thu Oct 10 01:30:35 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 9 Oct 2002 20:30:35 -0400 Subject: [Python-Dev] Where is this flag coming from? References: <15779.46954.672730.965163@gargle.gargle.HOWL> <15780.29440.641827.692637@gargle.gargle.HOWL> <200210092118.g99LICj26939@odiug.zope.com> Message-ID: <15780.51755.561632.395266@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> Maybe we should be more conservative and add this option only GvR> for Solaris? That was one suggestion I made when we talked about it last. Sounds like it's probably the right approach to whitelist the platforms we know are broken without it. I'm running a test on SF's compile farm Solaris machine now. GvR> How does one detect Solaris in setup.py? How about this GvR> patch (on top of the above one, not replacing it)? That's what I'm trying. GvR> I hope that Skip/Barry can test this on MacOS X and Martin on GvR> Solaris! I'll try OSX momentarily. GvR> Note: the 2.3 setup.py contains other uses of GvR> runtime_library_dirs, which probably need to be treated the GvR> same way. But maybe not. If you install any modern BerkeleyDB from source it always sticks it in a non-standard location. Hmm, maybe a blacklist is more useful for this one? Jeez this stuff sucks. GvR> Skip, can you cook up patches there? (The 2.2.2 GvR> fix for ssl also needs to be forward-ported!) Right, sorry, I'll do that if Guido's changes pans out for Solaris, Linux, and OSX. (Side note: I think I will have a few more minor patches for the email package, which I'll try to get into 2.2.2 tonight, if my cold will lay low for a few hours.) -Barry From barry@python.org Thu Oct 10 01:45:38 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 9 Oct 2002 20:45:38 -0400 Subject: [Python-Dev] Where is this flag coming from? References: <15779.46954.672730.965163@gargle.gargle.HOWL> <15780.29440.641827.692637@gargle.gargle.HOWL> <200210092118.g99LICj26939@odiug.zope.com> <15780.51755.561632.395266@gargle.gargle.HOWL> Message-ID: <15780.52658.412013.959026@gargle.gargle.HOWL> >>>>> "BAW" == Barry A Warsaw writes: GvR> I hope that Skip/Barry can test this on MacOS X and Martin on GvR> Solaris! BAW> I'll try OSX momentarily. The patch seems fine on RH7.3, and Solaris 8. OSX build not yet done, but I suspect it'll work. If so, I'll commit the change to setup.py for 2.2.2 and forward port it to 2.3. -Barry From tim@multitalents.net Thu Oct 10 03:01:27 2002 From: tim@multitalents.net (Tim Rice) Date: Wed, 9 Oct 2002 19:01:27 -0700 (PDT) Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <200210092118.g99LICj26939@odiug.zope.com> Message-ID: On Wed, 9 Oct 2002, Guido van Rossum wrote: > [Martin] > > For 2.2.2, backing out any setup.py changes to add > > runtime_library_dirs should be taking into consideration. > > This sounds like a conservative solution. There is exactly one such > option, for ssl. The checkin message: > [snip] > + runtime_library_dirs = ssl_libs, The only real problem is -R seems to be added for all unix platforms. This change would benefit all my platforms if it handled -R properly In OpenSSH we only use -R on solaris and freebsd. Perhaps runtime_library_dir_option() in Lib/distutils/unixccompiler.py needs some work. -- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net From barry@python.org Thu Oct 10 03:09:33 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 9 Oct 2002 22:09:33 -0400 Subject: [Python-Dev] Where is this flag coming from? References: <200210092118.g99LICj26939@odiug.zope.com> Message-ID: <15780.57693.291891.342258@gargle.gargle.HOWL> >>>>> "TR" == Tim Rice writes: TR> The only real problem is -R seems to be added for all unix TR> platforms. This change would benefit all my platforms if it TR> handled -R properly In OpenSSH we only use -R on solaris and TR> freebsd. Perhaps runtime_library_dir_option() in TR> Lib/distutils/unixccompiler.py needs some work. Anybody care to try a build on SF's freebsd system? If not, I'll try it after the Solaris build finishes. -Barry From tim@multitalents.net Thu Oct 10 03:25:15 2002 From: tim@multitalents.net (Tim Rice) Date: Wed, 9 Oct 2002 19:25:15 -0700 (PDT) Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <15780.57693.291891.342258@gargle.gargle.HOWL> Message-ID: On Wed, 9 Oct 2002, Barry A. Warsaw wrote: > > >>>>> "TR" == Tim Rice writes: > > TR> The only real problem is -R seems to be added for all unix > TR> platforms. This change would benefit all my platforms if it > TR> handled -R properly In OpenSSH we only use -R on solaris and > TR> freebsd. Perhaps runtime_library_dir_option() in > TR> Lib/distutils/unixccompiler.py needs some work. > I just tried Guido's setup.py patch on SCO Open Server and it works there. No more -R. :-) Am I out to lunch on the idea that the logic should be in Lib/distutils/unixccompiler.py where the -R is added? > Anybody care to try a build on SF's freebsd system? If not, I'll try > it after the Solaris build finishes. > > -Barry > -- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net From barry@python.org Thu Oct 10 03:35:42 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 9 Oct 2002 22:35:42 -0400 Subject: [Python-Dev] Where is this flag coming from? References: <15780.57693.291891.342258@gargle.gargle.HOWL> Message-ID: <15780.59262.401827.568672@gargle.gargle.HOWL> >>>>> "TR" == Tim Rice writes: TR> I just tried Guido's setup.py patch on SCO Open Server TR> and it works there. No more -R. :-) Cool. I think we're squared away for Python 2.2.2. TR> Am I out to lunch on the idea that the logic should be in TR> Lib/distutils/unixccompiler.py where the -R is added? That's what Skip checked in but only for Python 2.3 it seems. I'm a little confused now about what should be ported from 2.2.2 to 2.3, but maybe I'm still foggied headed from my cold. The SF freebsd build of 2.2.2 seemed to work fine, except that test_sax failed for reasons I don't know and don't have the energy right now to investigate. 2.2.2 works for me on all of RH7.3, Solaris 8, and OSX 10.2 (except for test_locale on the latter which has already been brought up). Python 2.3cvs is working fine for me on RH7.3. The Solaris 8 and OSX builds are still running, and I can't do the freebsd build until the Solaris one ends (well, I could, but it's easier to just wait). -Barry From tim@multitalents.net Thu Oct 10 05:26:33 2002 From: tim@multitalents.net (Tim Rice) Date: Wed, 9 Oct 2002 21:26:33 -0700 (PDT) Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <15780.59262.401827.568672@gargle.gargle.HOWL> Message-ID: On Wed, 9 Oct 2002, Barry A. Warsaw wrote: > > >>>>> "TR" == Tim Rice writes: > > TR> I just tried Guido's setup.py patch on SCO Open Server > TR> and it works there. No more -R. :-) > > Cool. I think we're squared away for Python 2.2.2. > > TR> Am I out to lunch on the idea that the logic should be in > TR> Lib/distutils/unixccompiler.py where the -R is added? > > That's what Skip checked in but only for Python 2.3 it seems. The runtime_library_dir_option function seems to be identical in both 2.3 and 2.2.2 > I'm a little confused now about what should be ported from 2.2.2 to > 2.3, but maybe I'm still foggied headed from my cold. > [snip] > -Barry > -- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net From mhammond@skippinet.com.au Thu Oct 10 05:42:16 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 10 Oct 2002 14:42:16 +1000 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib ntpath.py,1.50,1.51 In-Reply-To: <2md6qjd9zh.fsf@starship.python.net> Message-ID: [Tim in a checkin message] >> isn't anyone in PythonDevLand running CVS on Linux?!). > > Yes, but I do sleep occasionally... And my Linux box was stolen! New-one-supposed-to-be-here-yesterday--damn-dodgy-retailers ly Mark. From ishimoto@gembook.org Thu Oct 10 06:10:01 2002 From: ishimoto@gembook.org (Atsuo Ishimoto) Date: Thu, 10 Oct 2002 14:10:01 +0900 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: <200210092107.g99L7XA26891@odiug.zope.com> References: <200210092107.g99L7XA26891@odiug.zope.com> Message-ID: <20021010114451.7791.ISHIMOTO@gembook.org> Hello, On Wed, 09 Oct 2002 17:07:33 -0400 Guido van Rossum wrote: > Well, if you *want* to see the hex codes for all non-ASCII characters, > repr() used to be your friend. No more. If you *want* to see the > printable characters, you could always use print. I'm happy with this. I'm distributing modified version of Python Win32 installer at http://www.python.jp/Zope/download/pythonjpdist. This version of Python contains similar modifications for Japanese ShiftJIS users. But this patch has one problem. Because result of repr() depends on locale setting, we cannot assume text-form pickle could be restored everywhere. For example, under Japanese ShiftJIS locale, >>> s = '\x83\x5c' # This is a multi-byte character, third letter of "Python" >>> # in Japanese. Note that trailing character is '\' >>> >>> pickle.dump(s, f) I assume CVS version of Python fails to load this pickled object because backslash followed by quote is illegal. This problem may happens for Japanese ShiftJIS encoding, but I don't know whether there are another encodings causes same problem or not. I think this is not a major problem since we can avoid this by using binary form pickle, or using Unicode for text form pickle. But to eliminate this problem entirely, Python can have another slot to get a string representation of object, may be named tp_dumps. tp_dumps always returns hex codes for codes for all non-ASCII characters and is called whenever valid Python string literals are required. Regards, -------------------------- Atsuo Ishimoto ishimoto@gembook.jp Homepage:http://www.gembook.jp From perky@fallin.lv Thu Oct 10 07:42:37 2002 From: perky@fallin.lv (Hye-Shik Chang) Date: Thu, 10 Oct 2002 15:42:37 +0900 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: <20021010114451.7791.ISHIMOTO@gembook.org> References: <200210092107.g99L7XA26891@odiug.zope.com> <20021010114451.7791.ISHIMOTO@gembook.org> Message-ID: <20021010064237.GA8526@fallin.lv> Hi! On Thu, Oct 10, 2002 at 02:10:01PM +0900, Atsuo Ishimoto wrote: > Hello, > > On Wed, 09 Oct 2002 17:07:33 -0400 > Guido van Rossum wrote: > > > Well, if you *want* to see the hex codes for all non-ASCII characters, > > repr() used to be your friend. No more. If you *want* to see the > > printable characters, you could always use print. > > I'm happy with this. I'm distributing modified version of Python Win32 > installer at http://www.python.jp/Zope/download/pythonjpdist. This > version of Python contains similar modifications for Japanese ShiftJIS > users. > > But this patch has one problem. Because result of repr() depends on > locale setting, we cannot assume text-form pickle could be restored > everywhere. For example, under Japanese ShiftJIS locale, I realized that string_repr's depending on locale can be a problem maker for many unexpected situations. What I wanted in this patch is just to see _real_ string even in lists or dictionaries. I and CJKV users may feel happy even without string_repr locale patch. Regards, -- Hye-Shik Chang Yonsei University, Seoul ^D From ishimoto@gembook.org Thu Oct 10 09:28:29 2002 From: ishimoto@gembook.org (Atsuo Ishimoto) Date: Thu, 10 Oct 2002 17:28:29 +0900 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: <20021010064237.GA8526@fallin.lv> References: <20021010114451.7791.ISHIMOTO@gembook.org> <20021010064237.GA8526@fallin.lv> Message-ID: <20021010165744.7795.ISHIMOTO@gembook.org> Hi, Hye-Shik Chang. Thank you for submitting nice patch. On Thu, 10 Oct 2002 15:42:37 +0900 Hye-Shik Chang wrote: > I and CJKV users may feel happy even without string_repr locale patch. But this doesn't work for Idle or PythonWin. If sys.stdout is not real File object, PyFile_WriteObject() calls PyObject_Str() to get string to print. On printing list or dictionary, this will generate repr()ed string. Regards, -------------------------- Atsuo Ishimoto ishimoto@gembook.org Homepage:http://www.gembook.jp From Armin Rigo Thu Oct 10 09:52:19 2002 From: Armin Rigo (Armin Rigo) Date: Thu, 10 Oct 2002 10:52:19 +0200 (CEST) Subject: [Python-Dev] Set-next-statement in Python debuggers In-Reply-To: <200210091555.g99FtD011873@odiug.zope.com> Message-ID: Hello Guido, On Wed, 9 Oct 2002, Guido van Rossum wrote: > [Armin] > > You can already crash the interpreter with pure Python code, for example > > via the new.code() constructor or by writing crappy .pyc files. > > Yes, and I occasionally lose sleep over those. I definitely don't > want to add more loopholes like that, and I'd like to fix those. Ok. I already more or less committed to have a serious look over all the existing overflow problems that could cause a core dump (http://www.python.org/sf/618623). I may add in my to-do list the checks that no boggy data can crash unmarshall (after a first survey of the code I'd say that such data might exist; e.g. r_string() has a memcpy() that could be called with a negative size). More importantly, if you think that there is use for a complete "code object checker" that could catch any code doing silly things, then I'd love to contribute. Note however that this is something that will have to be kept closely in sync with future compiler extensions. A bientot, Armin. From mwh@python.net Thu Oct 10 09:55:17 2002 From: mwh@python.net (Michael Hudson) Date: 10 Oct 2002 09:55:17 +0100 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: Tim Rice's message of "Wed, 9 Oct 2002 19:25:15 -0700 (PDT)" References: Message-ID: <2m3crey8p6.fsf@starship.python.net> Tim Rice writes: > Am I out to lunch on the idea that the logic should be in > Lib/distutils/unixccompiler.py where the -R is added? No. The code we have right now is dubious, to say the least. Cheers, M. -- ... so the notion that it is meaningful to pass pointers to memory objects into which any random function may write random values without having a clue where they point, has _not_ been debunked as the sheer idiocy it really is. -- Erik Naggum, comp.lang.lisp From mwh@python.net Thu Oct 10 09:57:46 2002 From: mwh@python.net (Michael Hudson) Date: 10 Oct 2002 09:57:46 +0100 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 In-Reply-To: barry@python.org's message of "Wed, 9 Oct 2002 15:14:20 -0400" References: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> <15779.28468.831725.1313@gargle.gargle.HOWL> <2mfzvgnefw.fsf@starship.python.net> <15780.32780.635036.756155@gargle.gargle.HOWL> Message-ID: <2mzntmwu0l.fsf@starship.python.net> barry@python.org (Barry A. Warsaw) writes: > >>>>> "MH" == Michael Hudson writes: > > >> "JJ" == Jack Jansen writes: > >> JJ> Aside from the usual suspects I have three tests failing in > >> JJ> MacPython 2.2.2: test_atexit, test_email and test_httplib. > >> Oops, looks like I'm also getting a segfault in test_re. > > MH> This is normal; run "ulimit -s 2048" before running the test > MH> suite. > > Thanks, that works. Would it make sense to add this to the test suite > some how, either by adding a ulimit call to the makefile, or adding > some resource.setrlimit() call to site.py or test_{,s}re.py? There's a patch of sf to do this. It's probably a good idea (I forget to do this fairly often). Or maybe it could be in Modules/main.c:main()... Cheers, M. -- 39. Re graphics: A picture is worth 10K words - but only those to describe the picture. Hardly any sets of 10K words can be adequately described with pictures. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From mwh@python.net Thu Oct 10 10:00:44 2002 From: mwh@python.net (Michael Hudson) Date: 10 Oct 2002 10:00:44 +0100 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 In-Reply-To: "Neil Hodgson"'s message of "Thu, 10 Oct 2002 08:04:04 +1000" References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <200210071220.g97CKRG06496@pcp02138704pcs.reston01.va.comcast.net> <8765wbak3n.fsf@tleepslib.sk.tsukuba.ac.jp> <005001c26fdf$c8609c40$3da48490@neil> Message-ID: <2mwuoqwtvn.fsf@starship.python.net> "Neil Hodgson" writes: > Stephen J. Turnbull: > > > No current XEmacs recognizes cookies in the second line. I tried it, > > it broke, I backed it out, had a bad attack of day job, and forgot to > > fix the patch. :-( > > Where did the coding cookie originally come from? It seems to have > arrived in Python from somewhere else, probably Emacs but I couldn't find a > reference in PEP-263. >From the FSF Emacs 21.1 info file (node "Recognizing Coding Systems"): You can specify the coding system for a particular file using the `-*-...-*-' construct at the beginning of a file, or a local variables list at the end (*note File Variables::). You do this by defining a value for the "variable" named `coding'. Emacs does not really have a variable `coding'; instead of setting a variable, this uses the specified coding system for the file. For example, `-*-mode: C; coding: latin-1;-*-' specifies use of the Latin-1 coding system, as well as C mode. When you specify the coding explicitly in the file, that overrides `file-coding-system-alist'. Not sure if this existed in 20.x. I think the emacs manual is online somewhere, but not sure where. Cheers, M. -- For their next act, they'll no doubt be buying a firewall running under NT, which makes about as much sense as building a prison out of meringue. -- -:Tanuki:- -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html From fredrik@pythonware.com Thu Oct 10 10:13:07 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 10 Oct 2002 11:13:07 +0200 Subject: [Python-Dev] Non-ASCII characters in test_pep277.py in 2.3 References: <200210062218.g96MItv30307@pcp02138704pcs.reston01.va.comcast.net> <200210071220.g97CKRG06496@pcp02138704pcs.reston01.va.comcast.net> <8765wbak3n.fsf@tleepslib.sk.tsukuba.ac.jp> <005001c26fdf$c8609c40$3da48490@neil> <2mwuoqwtvn.fsf@starship.python.net> Message-ID: <019601c2703d$40c58e40$ced241d5@hagrid> michael wrote: > >From the FSF Emacs 21.1 info file (node "Recognizing Coding Systems"): > > You can specify the coding system for a particular file... > > I think the emacs manual is online somewhere, but not sure where. tried googling for "You can specify the coding system for a particular file"? here's the "I feel lucky" hit: http://www.delorie.com/gnu/docs/emacs/emacs_195.html From mwh@python.net Thu Oct 10 10:24:34 2002 From: mwh@python.net (Michael Hudson) Date: 10 Oct 2002 10:24:34 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib os.py,1.62,1.63 dospath.py,1.30,NONE In-Reply-To: loewis@users.sourceforge.net's message of "Wed, 09 Oct 2002 10:23:32 -0700" References: Message-ID: <2mvg4aeje5.fsf@starship.python.net> loewis@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Lib > In directory usw-pr-cvs1:/tmp/cvs-serv24338 > > Modified Files: > os.py > Removed Files: > dospath.py > Log Message: > Remove more DOS support. I think you should remove test_dospath & friends, too? Cheers, M. From guido@python.org Thu Oct 10 13:05:26 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 10 Oct 2002 08:05:26 -0400 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: Your message of "Wed, 09 Oct 2002 19:25:15 PDT." References: Message-ID: <200210101205.g9AC5Qm06716@pcp02138704pcs.reston01.va.comcast.net> > I just tried Guido's setup.py patch on SCO Open Server > and it works there. No more -R. :-) > > Am I out to lunch on the idea that the logic should be in > Lib/distutils/unixccompiler.py where the -R is added? Well, if we knew what to do on other systems instead of -R, that would help... I'm curious if perhaps Apple's loader (which started this) interprets -L as meaning both -L and -R in traditional Unix? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Oct 10 13:21:30 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 10 Oct 2002 08:21:30 -0400 Subject: [Python-Dev] Set-next-statement in Python debuggers In-Reply-To: Your message of "Thu, 10 Oct 2002 10:52:19 +0200." References: Message-ID: <200210101221.g9ACLVL06906@pcp02138704pcs.reston01.va.comcast.net> > > [Armin] > > > You can already crash the interpreter with pure Python code, for example > > > via the new.code() constructor or by writing crappy .pyc files. [Guido] > > Yes, and I occasionally lose sleep over those. I definitely don't > > want to add more loopholes like that, and I'd like to fix those. [Armin] > Ok. I already more or less committed to have a serious look over > all the existing overflow problems that could cause a core dump > (http://www.python.org/sf/618623). I may add in my to-do list the > checks that no boggy data can crash unmarshall (after a first survey > of the code I'd say that such data might exist; e.g. r_string() has > a memcpy() that could be called with a negative size). Fixes for this and similar problems will be most welcome! > More importantly, if you think that there is use for a complete > "code object checker" that could catch any code doing silly things, > then I'd love to contribute. Note however that this is something > that will have to be kept closely in sync with future compiler > extensions. Yes, the fact that our VM isn't standardized makes this a bit tricky. But I think it ought to be done. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Oct 10 13:59:26 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 10 Oct 2002 08:59:26 -0400 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: Your message of "Thu, 10 Oct 2002 14:10:01 +0900." <20021010114451.7791.ISHIMOTO@gembook.org> References: <200210092107.g99L7XA26891@odiug.zope.com> <20021010114451.7791.ISHIMOTO@gembook.org> Message-ID: <200210101259.g9ACxVW07997@pcp02138704pcs.reston01.va.comcast.net> [Guido] > > Well, if you *want* to see the hex codes for all non-ASCII characters, > > repr() used to be your friend. No more. If you *want* to see the > > printable characters, you could always use print. [Atsuo Ishimoto] > I'm happy with this. "This" was ambiguous. Are you happy with what's in current CVS, or with the old repr()? > I'm distributing modified version of Python Win32 > installer at http://www.python.jp/Zope/download/pythonjpdist. This > version of Python contains similar modifications for Japanese ShiftJIS > users. > > But this patch has one problem. Because result of repr() depends on > locale setting, we cannot assume text-form pickle could be restored > everywhere. For example, under Japanese ShiftJIS locale, > > >>> s = '\x83\x5c' # This is a multi-byte character, third letter of "Python" > >>> # in Japanese. Note that trailing character is '\' > >>> > >>> pickle.dump(s, f) > > I assume CVS version of Python fails to load this pickled object because > backslash followed by quote is illegal. This problem may happens for > Japanese ShiftJIS encoding, but I don't know whether there are another > encodings causes same problem or not. I tried this, and I could not find any problems with the resulting pickle. The pickle looks like this: "S'\\x83\\\\'\np0\n." I couldn't get this to fail loading in Python 2.1, 2.2 or 2.3 (CVS); I tried both pickle and cPickle. > I think this is not a major problem since we can avoid this by using > binary form pickle, or using Unicode for text form pickle. But to > eliminate this problem entirely, Python can have another slot to get a > string representation of object, may be named tp_dumps. > tp_dumps always returns hex codes for codes for all non-ASCII characters > and is called whenever valid Python string literals are required. I don't think this particular issue (pickling) is a problem. But I *do* continue to worry that making repr() depend on the locale may be a bigger problem than what it attempts to solve. [Hye-Shik Chang] > I realized that string_repr's depending on locale can be a problem > maker for many unexpected situations. What I wanted in this patch is > just to see _real_ string even in lists or dictionaries. > I and CJKV users may feel happy even without string_repr locale patch. I'm not sure I follow. What is the alternative that you propose? --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Thu Oct 10 14:04:22 2002 From: barry@python.org (Barry A. Warsaw) Date: Thu, 10 Oct 2002 09:04:22 -0400 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 References: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> <15779.28468.831725.1313@gargle.gargle.HOWL> <2mfzvgnefw.fsf@starship.python.net> <15780.32780.635036.756155@gargle.gargle.HOWL> <2mzntmwu0l.fsf@starship.python.net> Message-ID: <15781.31446.445161.102321@gargle.gargle.HOWL> >>>>> "MH" == Michael Hudson writes: MH> There's a patch of sf to do this. It's probably a good idea MH> (I forget to do this fairly often). Or maybe it could be in MH> Modules/main.c:main()... Do you mean #580869? It's not really much of a patch, e.g. I can't tell if Steve actually tried that or what file he put it in. I've assigned that PR to myself and will look into it for Python 2.3. Skip or Jack can feel free to steal it from me if they want to raise the urgency on it. -Barry From Jack.Jansen@cwi.nl Thu Oct 10 15:24:21 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Thu, 10 Oct 2002 16:24:21 +0200 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <200210101205.g9AC5Qm06716@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Thursday, Oct 10, 2002, at 14:05 Europe/Amsterdam, Guido van Rossum wrote: > > I'm curious if perhaps Apple's loader (which started this) interprets > -L as meaning both -L and -R in traditional Unix? For frameworks that is what happens, and while I haven't tried it for normal shared libraries the documentation suggest that this is indeed the case. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From list-python@ccraig.org Thu Oct 10 16:19:47 2002 From: list-python@ccraig.org (Christopher A. Craig) Date: 10 Oct 2002 11:19:47 -0400 Subject: [Python-Dev] Re: PEP239 (Rational Numbers) Reference Implementation and new issues In-Reply-To: References: Message-ID: Tim Peters writes: > If you haven't yet, you should strive to understand Moshe Zadka's prototype > rational implementation in Python CVS nondist/sandbox/ (which has all the > approximation functions you could need). Unfortunately, the SourceForge > Python Numerics list: > > https://sourceforge.net/mailarchive/forum.php?forum_id=2280 > > appears to have lost almost all the voluminous discussions that went into > that, retaining mostly a random sampling of spam. I have read the module. I really liked his (your?) trim code and I used that algorithm in cRat and my rational patch. I don't think I'd use any of that for generating decimal (or binary) float approximations, though. Trim introduces some error into the calculation which is then furthered by the float division. You can provably generate the closest scaled long of a given size (and thus hopefully the closest float) with one shift and one long integer division. -- Christopher A. Craig "It mearly pleases me to behave in a certain way to what appears to be a cat" -- The Ruler of the Universe (The Restaurant at the End of the Universe) From skip@pobox.com Thu Oct 10 17:09:14 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 10 Oct 2002 11:09:14 -0500 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 In-Reply-To: <15781.31446.445161.102321@gargle.gargle.HOWL> References: <7936B11E-DB07-11D6-B250-003065517236@oratrix.com> <15779.28468.831725.1313@gargle.gargle.HOWL> <2mfzvgnefw.fsf@starship.python.net> <15780.32780.635036.756155@gargle.gargle.HOWL> <2mzntmwu0l.fsf@starship.python.net> <15781.31446.445161.102321@gargle.gargle.HOWL> Message-ID: <15781.42538.323678.806285@montanaro.dyndns.org> BAW> Do you mean #580869? I gave it a try, but was unable to raise my stack size. I attached a file to the patch containing my change to Lib/test/regrtest.py. Skip From skip@pobox.com Thu Oct 10 04:25:18 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 9 Oct 2002 22:25:18 -0500 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: References: <15780.57693.291891.342258@gargle.gargle.HOWL> Message-ID: <15780.62238.554661.203642@montanaro.dyndns.org> Tim> Am I out to lunch on the idea that the logic should be in Tim> Lib/distutils/unixccompiler.py where the -R is added? I don't think so. How would you adjust it to work for SCO? Skip From skip@pobox.com Thu Oct 10 15:28:32 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 10 Oct 2002 09:28:32 -0500 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <200210101205.g9AC5Qm06716@pcp02138704pcs.reston01.va.comcast.net> References: <200210101205.g9AC5Qm06716@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15781.36496.719015.157394@montanaro.dyndns.org> Guido> I'm curious if perhaps Apple's loader (which started this) Guido> interprets -L as meaning both -L and -R in traditional Unix? Damned if I could tell by reading the ld(1) manpage. The change I checked in to unixccompiler.py returns -L only because I figured "-L" would be innocuous, not because I expected it to have "-R" semantics. I'll ask around on the Pythonmac-SIG. Skip From skip@pobox.com Thu Oct 10 04:08:34 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 9 Oct 2002 22:08:34 -0500 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <200210091536.g99FaVa11272@odiug.zope.com> References: <15779.41332.251210.360418@montanaro.dyndns.org> <15779.46954.672730.965163@gargle.gargle.HOWL> <200210091536.g99FaVa11272@odiug.zope.com> Message-ID: <15780.61234.606750.832257@montanaro.dyndns.org> >> Shouldn't distutils be taught what to do with runtime_library_dirs >> for SCO and MacOSX? Guido> My thoughts exactly. Given that MacOSX's linker doesn't seem to understand -R at all (no matter how you deliver it), I adjusted the code in runtime_library_dir_option (in unixccompiler.py) to return "-L" + dir instead of either of the "-R" possibilities. This is probably not as good as it could be, but I couldn't tease anything from the ld(1) man page which looked to me like an equivalent to "-R" or LD_RUN_PATH. Perhaps someone with more experience with MacOSX or link editors in general would like to take a peek at the problem. If so, let me know and I'll be happy to shoot you a copy of the ld(1) man page. Skip From skip@pobox.com Thu Oct 10 04:38:02 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 9 Oct 2002 22:38:02 -0500 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <15780.59262.401827.568672@gargle.gargle.HOWL> References: <15780.57693.291891.342258@gargle.gargle.HOWL> <15780.59262.401827.568672@gargle.gargle.HOWL> Message-ID: <15780.63002.206139.926109@montanaro.dyndns.org> TR> Am I out to lunch on the idea that the logic should be in TR> Lib/distutils/unixccompiler.py where the -R is added? BAW> That's what Skip checked in but only for Python 2.3 it seems. I'm BAW> a little confused now about what should be ported from 2.2.2 to BAW> 2.3, but maybe I'm still foggied headed from my cold. I thought this was only a 2.3 thing. I just tried on the release22_maint CVS branch. The bsddb module compiled and linked fine, but I don't know why. Oh, wait a minute... The release22_maint branch doesn't have the bsddb bells and whistles I added to setup.py a couple months ago. Consequently, it's picking up the 1.85 version of the include file and library (which, I will remind people has known breakage). Should I backport those changes? I think what's in 2.3's setup.py is much more robust, but if people have been getting Berkeley DB 1.85 libraries in 2.2.1 and don't get a clean compile in 2.2.2 without editing setup.py because 1.85 is all they have available, that might be construed as "more than a bugfix". Let me know what you want to do in this regard Guido. Skip From skip@pobox.com Thu Oct 10 17:24:48 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 10 Oct 2002 11:24:48 -0500 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: References: <15780.57693.291891.342258@gargle.gargle.HOWL> <15780.59262.401827.568672@gargle.gargle.HOWL> Message-ID: <15781.43472.667036.609668@montanaro.dyndns.org> Martin> It seems that Skip chose the blacklist approach (aka the Martin> I-care-only-about-MacOS approach). Skip's approach is probably Martin> overkill, since there will be already -L options for all those Martin> directories. It appears that OS X records the full path of the Martin> library in the binary, so the notion of a runtime path is Martin> irrelevant. I didn't want to return "" because that would require me to filter it out of the argument list elsewhere. "-L" + dir seemed like the most innocuous change. Skip From guido@python.org Thu Oct 10 17:34:58 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 10 Oct 2002 12:34:58 -0400 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: Your message of "Thu, 10 Oct 2002 09:28:32 CDT." <15781.36496.719015.157394@montanaro.dyndns.org> References: <200210101205.g9AC5Qm06716@pcp02138704pcs.reston01.va.comcast.net> <15781.36496.719015.157394@montanaro.dyndns.org> Message-ID: <200210101634.g9AGYwn14473@odiug.zope.com> > Guido> I'm curious if perhaps Apple's loader (which started this) > Guido> interprets -L as meaning both -L and -R in traditional Unix? > > Damned if I could tell by reading the ld(1) manpage. The change I > checked in to unixccompiler.py returns -L only because I figured > "-L" would be innocuous, not because I expected it to have > "-R" semantics. Why did you check it in to Python 2.3 but not to 2.2.2? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Oct 10 17:36:32 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 10 Oct 2002 12:36:32 -0400 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: Your message of "Wed, 09 Oct 2002 22:38:02 CDT." <15780.63002.206139.926109@montanaro.dyndns.org> References: <15780.57693.291891.342258@gargle.gargle.HOWL> <15780.59262.401827.568672@gargle.gargle.HOWL> <15780.63002.206139.926109@montanaro.dyndns.org> Message-ID: <200210101636.g9AGaWM14486@odiug.zope.com> > I thought this was only a 2.3 thing. But this thread started with you reporting a 2.2.2 problem trying to compile SSL! > I just tried on the release22_maint CVS branch. The bsddb module > compiled and linked fine, but I don't know why. Oh, wait a > minute... The release22_maint branch doesn't have the bsddb bells > and whistles I added to setup.py a couple months ago. Consequently, > it's picking up the 1.85 version of the include file and library > (which, I will remind people has known breakage). > > Should I backport those changes? I think what's in 2.3's setup.py > is much more robust, but if people have been getting Berkeley DB > 1.85 libraries in 2.2.1 and don't get a clean compile in 2.2.2 > without editing setup.py because 1.85 is all they have available, > that might be construed as "more than a bugfix". > > Let me know what you want to do in this regard Guido. I think it's too precarious to fix in 2.2.2. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Thu Oct 10 18:29:46 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 10 Oct 2002 12:29:46 -0500 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <200210101634.g9AGYwn14473@odiug.zope.com> References: <200210101205.g9AC5Qm06716@pcp02138704pcs.reston01.va.comcast.net> <15781.36496.719015.157394@montanaro.dyndns.org> <200210101634.g9AGYwn14473@odiug.zope.com> Message-ID: <15781.47370.595649.754498@montanaro.dyndns.org> >>>>> "Guido" == Guido van Rossum writes: Guido> I'm curious if perhaps Apple's loader (which started this) Guido> interprets -L as meaning both -L and -R in traditional Unix? >> >> Damned if I could tell by reading the ld(1) manpage. The change I >> checked in to unixccompiler.py returns -L only because I figured >> "-L" would be innocuous, not because I expected it to have >> "-R" semantics. Guido> Why did you check it in to Python 2.3 but not to 2.2.2? Because I couldn't build bsddb on 2.3 without it. Skip From skip@pobox.com Thu Oct 10 18:30:58 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 10 Oct 2002 12:30:58 -0500 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <200210101636.g9AGaWM14486@odiug.zope.com> References: <15780.57693.291891.342258@gargle.gargle.HOWL> <15780.59262.401827.568672@gargle.gargle.HOWL> <15780.63002.206139.926109@montanaro.dyndns.org> <200210101636.g9AGaWM14486@odiug.zope.com> Message-ID: <15781.47442.968831.672932@montanaro.dyndns.org> >> I thought this was only a 2.3 thing. Guido> But this thread started with you reporting a 2.2.2 problem trying Guido> to compile SSL! Nope, it was bsddb on 2.3. Someone else was talking about SSL, which I could generally care less about. I use bsddb all the time, however. Skip From sdm7g@mac.com Thu Oct 10 18:43:11 2002 From: sdm7g@mac.com (Steven Majewski) Date: Thu, 10 Oct 2002 13:43:11 -0400 Subject: [Python-Dev] resource.setrlimit on OSX [was:Tests failing in MacPython 2.2.2] In-Reply-To: <15781.42538.323678.806285@montanaro.dyndns.org> Message-ID: On Thursday, October 10, 2002, at 12:09 PM, Skip Montanaro wrote: > > BAW> Do you mean #580869? > > I gave it a try, but was unable to raise my stack size. I attached a file > to the patch containing my change to Lib/test/regrtest.py. > > Skip > Skip -- I saw the sourceforge notice and replied there before I saw this thread. Yes, that fix still works for me and No, I'm not running it as root, but I am still running 10.1.5, so if it's not working for you, maybe it's a 10.1 vs 10.2 problem. -- Steve From martin@v.loewis.de Thu Oct 10 19:12:10 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 10 Oct 2002 20:12:10 +0200 Subject: [Python-Dev] Where is this flag coming from? In-Reply-To: <15781.36496.719015.157394@montanaro.dyndns.org> References: <200210101205.g9AC5Qm06716@pcp02138704pcs.reston01.va.comcast.net> <15781.36496.719015.157394@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > Damned if I could tell by reading the ld(1) manpage. The change I checked > in to unixccompiler.py returns -L only because I figured "-L" would be > innocuous, not because I expected it to have "-R" semantics. Thanks to 'otool -L', you'll see that the complete path of the library is embedded into the executable. Regards, Martin From martin@v.loewis.de Thu Oct 10 19:25:21 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 10 Oct 2002 20:25:21 +0200 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib os.py,1.62,1.63 dospath.py,1.30,NONE In-Reply-To: <2mvg4aeje5.fsf@starship.python.net> References: <2mvg4aeje5.fsf@starship.python.net> Message-ID: Michael Hudson writes: > I think you should remove test_dospath & friends, too? Done for test_dospath, and documentation. What other friends did you have in mind? Regards, Martin From mwh@python.net Thu Oct 10 19:26:55 2002 From: mwh@python.net (Michael Hudson) Date: Thu, 10 Oct 2002 19:26:55 +0100 (BST) Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib os.py,1.62,1.63 dospath.py,1.30,NONE In-Reply-To: Message-ID: On 10 Oct 2002, Martin v. Loewis wrote: > Michael Hudson writes: > > > I think you should remove test_dospath & friends, too? > > Done for test_dospath, and documentation. What other friends did you > have in mind? I was mainly just being vague. I thought there was a test output file, but there wasn't/isn't. Cheers, M. From martin@v.loewis.de Thu Oct 10 19:29:27 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 10 Oct 2002 20:29:27 +0200 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: <200210101259.g9ACxVW07997@pcp02138704pcs.reston01.va.comcast.net> References: <200210092107.g99L7XA26891@odiug.zope.com> <20021010114451.7791.ISHIMOTO@gembook.org> <200210101259.g9ACxVW07997@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > > I realized that string_repr's depending on locale can be a problem > > maker for many unexpected situations. What I wanted in this patch is > > just to see _real_ string even in lists or dictionaries. > > I and CJKV users may feel happy even without string_repr locale patch. > > I'm not sure I follow. What is the alternative that you propose? I think Hye-Shik Chang would be happy if just tp_print acts according to the locale, and "most" containers support tp_print. Given that list, tuple, and dict have tp_print, no further action might be needed. I'm now convinced that having repr locale-specific is unacceptable. So I would back something out. However, I still sympathize with the original rationale, and I don't see a problem with tp_print differing from tp_repr. Regards, Martin From guido@python.org Thu Oct 10 19:52:02 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 10 Oct 2002 14:52:02 -0400 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: Your message of "Thu, 10 Oct 2002 20:29:27 +0200." References: <200210092107.g99L7XA26891@odiug.zope.com> <20021010114451.7791.ISHIMOTO@gembook.org> <200210101259.g9ACxVW07997@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210101852.g9AIq2k14887@odiug.zope.com> > > > I realized that string_repr's depending on locale can be a problem > > > maker for many unexpected situations. What I wanted in this patch is > > > just to see _real_ string even in lists or dictionaries. > > > I and CJKV users may feel happy even without string_repr locale patch. > > > > I'm not sure I follow. What is the alternative that you propose? > > I think Hye-Shik Chang would be happy if just tp_print acts according > to the locale, and "most" containers support tp_print. Given that > list, tuple, and dict have tp_print, no further action might be > needed. That should not be done. tp_print only gets invoked when sys.stdout is a real file; otherwise str() or repr() gets invoked. This should be transparent; tp_print should be an optimization only, not a way to get different semantics. It has already been mentioned that tp_print is not invoked when using print from IDLE. > I'm now convinced that having repr locale-specific is unacceptable. So > I would back something out. However, I still sympathize with the > original rationale, and I don't see a problem with tp_print differing > from tp_repr. I do, see above. Feel free to back out the multibyte-string patches. We have learned something. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Thu Oct 10 19:49:53 2002 From: barry@python.org (Barry A. Warsaw) Date: Thu, 10 Oct 2002 14:49:53 -0400 Subject: [Python-Dev] resource.setrlimit on OSX [was:Tests failing in MacPython 2.2.2] References: <15781.42538.323678.806285@montanaro.dyndns.org> Message-ID: <15781.52177.932856.1255@gargle.gargle.HOWL> >>>>> "SM" == Steven Majewski writes: SM> Yes, that fix still works for me and No, I'm not running it as SM> root, but I am still running 10.1.5, so if it's not working SM> for you, maybe it's a 10.1 vs 10.2 problem. Note that SourceForge's compile farm has some vintage 10.1 OSX server available. I've got 10.2 at home. I'll try it on both. -Barry From skip@pobox.com Thu Oct 10 20:27:27 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 10 Oct 2002 14:27:27 -0500 Subject: [Python-Dev] Re: resource.setrlimit on OSX [was:Tests failing in MacPython 2.2.2] In-Reply-To: References: <15781.42538.323678.806285@montanaro.dyndns.org> Message-ID: <15781.54431.709070.201671@montanaro.dyndns.org> Steve> Yes, that fix still works for me and No, I'm not running it as Steve> root, but I am still running 10.1.5, so if it's not working for Steve> you, maybe it's a 10.1 vs 10.2 problem. Alas, from what I've seen on the pythonmac-sig, this seems to be a common theme... Skip From richie@entrian.com Thu Oct 10 21:13:33 2002 From: richie@entrian.com (Richie Hindle) Date: Thu, 10 Oct 2002 21:13:33 +0100 Subject: [Python-Dev] Set-next-statement in Python debuggers In-Reply-To: <200210091555.g99FtD011873@odiug.zope.com> References: <200210091555.g99FtD011873@odiug.zope.com> Message-ID: Hi, [Guido] > Python should not include features (like writable code object > attributes) that can cause crashes when used inexpertly. If I do implement this, I'll do it in such a way as to make crashes impossible (my first suggestion said that crashes would be possible, but I realise that was a mistake). It'll verify that the new value of f_lasti is a valid bytecode position, and it'll either fix up the stack/blockstack or refuse to move the position in ways that would affect them. (Armin, I guess that won't restrict you because you're doing this from C code?) [Armin] > For safety I'd consider writing the > frame-object-modifying code in a C extension module, carefully > documented as "don't use this". [Guido] > That's a reasonable solution: f_lasti should be read-only from Python > code, but you can write an extension that can write it. I'd like to see pure Python debuggers (including pdb) have this feature. If it's safe to use, might it still be considered for inclusion? (I can't guarantee I'll have the time to do it, but I'll try.) I'm also coming to realise that it's not f_lasti I really want to change, but f_lineno. The line number is the 'unit' that debuggers use - a debugger would set f_lineno, and f_lasti would be calculated and changed to the appropriate value. I don't think that changes any of the arguments either way, except maybe to distance the idea a little from what Armin is doing. -- Richie Hindle richie@entrian.com From guido@python.org Thu Oct 10 21:32:01 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 10 Oct 2002 16:32:01 -0400 Subject: [Python-Dev] Set-next-statement in Python debuggers In-Reply-To: Your message of "Thu, 10 Oct 2002 21:13:33 BST." References: <200210091555.g99FtD011873@odiug.zope.com> Message-ID: <200210102032.g9AKW1K15947@odiug.zope.com> > [Guido] > > Python should not include features (like writable code object > > attributes) that can cause crashes when used inexpertly. > > If I do implement this, I'll do it in such a way as to make crashes > impossible (my first suggestion said that crashes would be possible, > but I realise that was a mistake). It'll verify that the new value > of f_lasti is a valid bytecode position, and it'll either fix up the > stack/blockstack or refuse to move the position in ways that would > affect them. (Armin, I guess that won't restrict you because you're > doing this from C code?) Great. > [Armin] > > For safety I'd consider writing the > > frame-object-modifying code in a C extension module, carefully > > documented as "don't use this". > > [Guido] > > That's a reasonable solution: f_lasti should be read-only from > > Python code, but you can write an extension that can write it. > > I'd like to see pure Python debuggers (including pdb) have this > feature. If it's safe to use, might it still be considered for > inclusion? (I can't guarantee I'll have the time to do it, but I'll > try.) Sure, *if* it's safe. > I'm also coming to realise that it's not f_lasti I really want to > change, but f_lineno. The line number is the 'unit' that debuggers > use - a debugger would set f_lineno, and f_lasti would be calculated > and changed to the appropriate value. I don't think that changes > any of the arguments either way, except maybe to distance the idea a > little from what Armin is doing. Sure. --Guido van Rossum (home page: http://www.python.org/~guido/) From Jack.Jansen@oratrix.com Thu Oct 10 21:41:51 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Thu, 10 Oct 2002 22:41:51 +0200 Subject: [Python-Dev] Re: resource.setrlimit on OSX [was:Tests failing in MacPython 2.2.2] In-Reply-To: <15781.54431.709070.201671@montanaro.dyndns.org> Message-ID: On donderdag, oktober 10, 2002, at 09:27 , Skip Montanaro wrote: > > Steve> Yes, that fix still works for me and No, I'm not > running it as > Steve> root, but I am still running 10.1.5, so if it's not > working for > Steve> you, maybe it's a 10.1 vs 10.2 problem. > > Alas, from what I've seen on the pythonmac-sig, this seems to > be a common > theme... Don't worry: the recent flurry of discussion is all about MacPython, not about unix-Python. Forward compatibility from 10.1 to 10.2 for unix programs is pretty good, the only problem I've seen (runtime-wise, that is) is with programs using curses. What *is* bothersome, though, is that you cannot easily create a program on 10.2 that is backwards compatible to 10.1. Now this is probably true on most Unix platforms, but Apple has all this wonderful versioning infrastructure for frameworks, and then they don't use it themselves:-( -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From guido@python.org Thu Oct 10 22:04:53 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 10 Oct 2002 17:04:53 -0400 Subject: [Python-Dev] tp_dictoffset calculation in 2.2.2 Message-ID: <200210102105.g9AL4rP16034@odiug.zope.com> Hi David and Kevin, I'd like to hear from you if a particular last-minute change to 2.2.2 would affect your code in any way. If you don't have time to read this whole message, can you please check if this patch to Python 2.2.2b1 (in CVS on the release22-maint branch) has any bad effects on your code? diff -r2.126.4.24 typeobject.c 2090c2090 < COPYSLOT(tp_dictoffset); --- > /*COPYSLOT(tp_dictoffset);*/ Background ---------- In Python 2.2, type objects have a tp_dictoffset which tells us where the instance dict pointer is in the instance object layout. This pointer is calculated in three ways. In precedence: 1. (Line 1113) For dynamically created types (i.e. created by class statements), if the new type has no __slots__, the dominant base class has a zero tp_dictoffset, and the dominant base doesn't have a custom tp_getattro, a __dict__ will be added to the instance layout, and its tp_dictoffset is calculated by taking the end of the base class instance struct (or in a more complicated way if tp_itemsize is nonzero). 2. (Line 1941) If the dominant base has a nonzero tp_dictoffset, copy the tp_dictoffset from the dominant base. 3. (Line 2090) The tp_dictoffset of the first non-dominant base class that has a nonzero tp_dictoffset is copied. Jeremy & I just went through a debugging session of some code he's trying to add to Zope3 where we found that rule #3 is evil. Simplified, we had the following situation: from some_extension import P class C(object): pass class D(C, P): pass Type P in has some unique properties: its tp_dictoffset is zero, and it has a custom tp_setattro. This means that neither rule #1 fires (custom tp_setattro) nor rule #2 (the dominant base has no tp_dictoffset). So the tp_dictoffset from class C is simply copied into class D. This offset is 12, but at that offset lives some unrelated field in the P instance layout! I believe that in cases where rule #3 triggers, it is *always* wrong: when rule #2 hasn't triggered, the dict offset in the dominant base class is different from the dict offset in the other base classes, but the instance layout is inherited only from the dominant base. But I can't prove this to myself with 100% satisfaction, so I'd like to hear some test results, before I check this in as part of 2.2.2 final. (The other COPYSLOT calls in the vicinity of the patch are also a bit suspect, but much less so, since they are function pointers rather than object offsets.) --Guido van Rossum (home page: http://www.python.org/~guido/) From Jack.Jansen@oratrix.com Thu Oct 10 22:07:26 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Thu, 10 Oct 2002 23:07:26 +0200 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 In-Reply-To: <15779.29749.886457.639973@gargle.gargle.HOWL> Message-ID: <47ACA202-DC94-11D6-9207-003065517236@oratrix.com> On woensdag, oktober 9, 2002, at 02:11 , Barry A. Warsaw wrote: > JJ> test_email: assertion failed, 2042 != 2. from line 1793 in > JJ> test_crlf_separation. Could this have something to do with > JJ> MacPython 2.2's "poor man's universal newlines"? (MacPython > JJ> 2.2.X will read both \r and \n as \n for files opened in text > JJ> mode, but not \r\n. This quick hack was put in to make 2.2 > JJ> usable on OSX). > > Re-reading this, are you saying that this failure is on MacOS9 only? Correct. Or, to be hyper-correct: it fails in MacPython 2.2.X on OSX as well, but not in unix-Python 2.2.X on OSX. > That test passes just fine on OSX 10.2 and the data looks just fine to > me, so this must be an OS9 failure only. > > Maybe the thing to do is to open msg_26.txt in binary mode, but only > on MacOS9? Doesn't work, now the assert just above that fails (the payload length is 0 in stead of 2), so I think this breaks the boundary separation or somesuch. I've added code to simply skip this test for MacPython 2.2.X, how do you feel about that? -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From Jack.Jansen@oratrix.com Thu Oct 10 22:09:12 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Thu, 10 Oct 2002 23:09:12 +0200 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 In-Reply-To: <200210090034.g990YTx26342@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <8667DDD1-DC94-11D6-9207-003065517236@oratrix.com> On woensdag, oktober 9, 2002, at 02:34 , Guido van Rossum wrote: >> The first one should be easy to fix: backport the revision 1.7 >> fix. Would that cause any problems on other platforms? > > No problemo. Ok, done. >> test_httplib: mismatch in lines 9-12 of expected output. It >> wanted two header: Set-Cookie: lines, but got nothing. > > Probably a line ending problem. I notice that those two lines end > with \r\n! I don't understand why the test uses \r\n. I propose to > change all the \r\n to \n in test_httplib.py and ditto in > output/test_httplib. Ok, done. I haven't checked it on unix yet, I'll do that shortly. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From jacobs@penguin.theopalgroup.com Thu Oct 10 22:43:34 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Thu, 10 Oct 2002 17:43:34 -0400 (EDT) Subject: [Python-Dev] tp_dictoffset calculation in 2.2.2 In-Reply-To: <200210102105.g9AL4rP16034@odiug.zope.com> Message-ID: On Thu, 10 Oct 2002, Guido van Rossum wrote: > I'd like to hear from you if a particular last-minute change to 2.2.2 > would affect your code in any way. If you don't have time to read > this whole message, can you please check if this patch to Python > 2.2.2b1 (in CVS on the release22-maint branch) has any bad effects on > your code? > > diff -r2.126.4.24 typeobject.c > 2090c2090 > < COPYSLOT(tp_dictoffset); > --- > > /*COPYSLOT(tp_dictoffset);*/ This doesn't seem to interact with any of my extension classes, or at least nothing in our test suite. I'll look into this more deeply over the next few days, though I agree that case #3 is almost always certain to do the wrong thing. Thanks for the heads-up, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From dave@boost-consulting.com Thu Oct 10 22:44:41 2002 From: dave@boost-consulting.com (David Abrahams) Date: 10 Oct 2002 17:44:41 -0400 Subject: [Python-Dev] Re: tp_dictoffset calculation in 2.2.2 In-Reply-To: <200210102105.g9AL4rP16034@odiug.zope.com> References: <200210102105.g9AL4rP16034@odiug.zope.com> Message-ID: Guido van Rossum writes: > Hi David and Kevin, > > I'd like to hear from you if a particular last-minute change to 2.2.2 > would affect your code in any way. If you don't have time to read > this whole message, can you please check if this patch to Python > 2.2.2b1 (in CVS on the release22-maint branch) has any bad effects on > your code? I'll try running my test suite against it now. -- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From barry@python.org Thu Oct 10 22:55:11 2002 From: barry@python.org (Barry A. Warsaw) Date: Thu, 10 Oct 2002 17:55:11 -0400 Subject: [Python-Dev] Tests failing in MacPython 2.2.2 References: <15779.29749.886457.639973@gargle.gargle.HOWL> <47ACA202-DC94-11D6-9207-003065517236@oratrix.com> Message-ID: <15781.63295.453362.50932@gargle.gargle.HOWL> >>>>> "JJ" == Jack Jansen writes: >> Re-reading this, are you saying that this failure is on MacOS9 >> only? JJ> Correct. Or, to be hyper-correct: it fails in MacPython 2.2.X JJ> on OSX as well, but not in unix-Python 2.2.X on OSX. Ok, cool. I'm of course only testing unix-Python on OSX. >> That test passes just fine on OSX 10.2 and the data looks just >> fine to me, so this must be an OS9 failure only. >> Maybe the thing to do is to open msg_26.txt in binary mode, but >> only on MacOS9? JJ> Doesn't work, now the assert just above that fails (the JJ> payload length is 0 in stead of 2), so I think this breaks the JJ> boundary separation or somesuch. I've added code to simply JJ> skip this test for MacPython 2.2.X, how do you feel about JJ> that? I'm happy with that for Python 2.2.2, since I'm convinced it's just a problem with the test suite. When we have more breathing room (i.e. Python 2.3) we can see if we can come up with something more useful. -Barry From dave@boost-consulting.com Thu Oct 10 23:08:54 2002 From: dave@boost-consulting.com (David Abrahams) Date: 10 Oct 2002 18:08:54 -0400 Subject: [Python-Dev] Re: tp_dictoffset calculation in 2.2.2 In-Reply-To: <200210102105.g9AL4rP16034@odiug.zope.com> References: <200210102105.g9AL4rP16034@odiug.zope.com> Message-ID: Guido van Rossum writes: All of my tests passed. Thanks for asking, though -- we're just about to release Boost.Python v2 (like, tomorrow!) -Dave -- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From neal@metaslash.com Thu Oct 10 23:59:54 2002 From: neal@metaslash.com (Neal Norwitz) Date: Thu, 10 Oct 2002 18:59:54 -0400 Subject: [Python-Dev] Multiple Inclusion of Header Files Message-ID: <3DA6066A.F349116F@metaslash.com> >From SF #607253, header file problems ( http://python.org/sf/607253 ) is there any reason why these files: graminit.h patchlevel.h pyconfig.h (aka pyconfig.h.in) don't protect against being included multiple times? ie, they don't have: #ifndef Py_FILE_H #define Py_FILE_H /* ... */ #endif /* !Py_FILE_H */ Neal From ishimoto@gembook.org Fri Oct 11 00:06:33 2002 From: ishimoto@gembook.org (Atsuo Ishimoto) Date: Fri, 11 Oct 2002 08:06:33 +0900 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: <200210101259.g9ACxVW07997@pcp02138704pcs.reston01.va.comcast.net> References: <20021010114451.7791.ISHIMOTO@gembook.org> <200210101259.g9ACxVW07997@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021011075000.E88A.ISHIMOTO@gembook.org> Hello, On Thu, 10 Oct 2002 08:59:26 -0400 Guido van Rossum wrote: > [Guido] > > > Well, if you *want* to see the hex codes for all non-ASCII characters, > > > repr() used to be your friend. No more. If you *want* to see the > > > printable characters, you could always use print. > > [Atsuo Ishimoto] > > I'm happy with this. > > "This" was ambiguous. Are you happy with what's in current CVS, or > with the old repr()? Sorry. I meant I'm happy with current CVS. > I tried this, and I could not find any problems with the resulting > pickle. The pickle looks like this: > > "S'\\x83\\\\'\np0\n." > > I couldn't get this to fail loading in Python 2.1, 2.2 or 2.3 (CVS); > I tried both pickle and cPickle. May be you tried with ascii or latin-1 locale. But with Japanese ShiftJIS encoding, you'll see the traceback on unpickling. I tried CVS version of python on Japanese edition of Win2000, I got: Traceback (most recent call last): File "", line 1, in ? File "C:\cygwin\home\ishimoto\wk\python\dist\src\lib\pickle.py", line 1071, in loads return Unpickler(file).load() File "C:\cygwin\home\ishimoto\wk\python\dist\src\lib\pickle.py", line 675, in load dispatch[key](self) File "C:\cygwin\home\ishimoto\wk\python\dist\src\lib\pickle.py", line 754, in load_string self.append(rep.decode("string-escape")) ValueError: Trailing \ in string -------------------------- Atsuo Ishimoto ishimoto@gembook.org Homepage:http://www.gembook.jp From guido@python.org Fri Oct 11 00:58:36 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 10 Oct 2002 19:58:36 -0400 Subject: [Python-Dev] Multiple Inclusion of Header Files In-Reply-To: Your message of "Thu, 10 Oct 2002 18:59:54 EDT." <3DA6066A.F349116F@metaslash.com> References: <3DA6066A.F349116F@metaslash.com> Message-ID: <200210102358.g9ANwa811524@pcp02138704pcs.reston01.va.comcast.net> > From SF #607253, header file problems > ( http://python.org/sf/607253 ) > is there any reason why these files: > graminit.h > patchlevel.h > pyconfig.h (aka pyconfig.h.in) > > don't protect against being included multiple times? > > ie, they don't have: > > #ifndef Py_FILE_H > #define Py_FILE_H > > /* ... */ > > #endif /* !Py_FILE_H */ Could've been an oversight. I notice that graminit.h and pyconfig.h are generated files. patchlevel.h was originally a one-liner. Is there anything that is gained by adding those? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 11 01:04:05 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 10 Oct 2002 20:04:05 -0400 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: Your message of "Fri, 11 Oct 2002 08:06:33 +0900." <20021011075000.E88A.ISHIMOTO@gembook.org> References: <20021010114451.7791.ISHIMOTO@gembook.org> <200210101259.g9ACxVW07997@pcp02138704pcs.reston01.va.comcast.net> <20021011075000.E88A.ISHIMOTO@gembook.org> Message-ID: <200210110004.g9B045011881@pcp02138704pcs.reston01.va.comcast.net> > > I tried this, and I could not find any problems with the resulting > > pickle. The pickle looks like this: > > > > "S'\\x83\\\\'\np0\n." > > > > I couldn't get this to fail loading in Python 2.1, 2.2 or 2.3 (CVS); > > I tried both pickle and cPickle. > > May be you tried with ascii or latin-1 locale. But with Japanese > ShiftJIS encoding, you'll see the traceback on unpickling. > > I tried CVS version of python on Japanese edition of Win2000, I got: > > Traceback (most recent call last): > File "", line 1, in ? > File "C:\cygwin\home\ishimoto\wk\python\dist\src\lib\pickle.py", line 1071, in loads > return Unpickler(file).load() > File "C:\cygwin\home\ishimoto\wk\python\dist\src\lib\pickle.py", line 675, in load > dispatch[key](self) > File "C:\cygwin\home\ishimoto\wk\python\dist\src\lib\pickle.py", line 754, in load_string > self.append(rep.decode("string-escape")) > ValueError: Trailing \ in string Oops. It looks like this is the nail in the coffin of the locale-specific repr(): a multibyte "character" may end in a backslash that is not escaped. Martin, I'm afraid there's nothing we can do except revert your multibyte character checkins... --Guido van Rossum (home page: http://www.python.org/~guido/) From pinard@iro.umontreal.ca Fri Oct 11 04:28:02 2002 From: pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois?= Pinard) Date: Thu, 10 Oct 2002 23:28:02 -0400 Subject: [Python-Dev] Re: Multiple Inclusion of Header Files In-Reply-To: <200210102358.g9ANwa811524@pcp02138704pcs.reston01.va.comcast.net> (Guido van Rossum's message of "Thu, 10 Oct 2002 19:58:36 -0400") References: <3DA6066A.F349116F@metaslash.com> <200210102358.g9ANwa811524@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido van Rossum] >> From SF #607253, header file problems >> ( http://python.org/sf/607253 ) >> is there any reason why these files: >> graminit.h >> patchlevel.h >> pyconfig.h (aka pyconfig.h.in) >> >> don't protect against being included multiple times? >> >> ie, they don't have: >> >> #ifndef Py_FILE_H >> #define Py_FILE_H >> >> /* ... */ >> >> #endif /* !Py_FILE_H */ > > Could've been an oversight. I notice that graminit.h and pyconfig.h > are generated files. patchlevel.h was originally a one-liner. > > Is there anything that is gained by adding those? I never have been comfortable with the trend of recent years about include files protecting themselves against multiple inclusion, which progressively replaced the previous trend about including more carefully than blindly. Granted, the wild variety of organisations for include files in various operating systems is a problem in itself, writing portably in that almost intractable mess is undoubtedly a challenge. Protection against multiple inclusion has been welcome in that area. However, this habit of protection is spreading (I might even have given into it myself) and I sometimes wonder if we do this is by mere mimetism, or for any good reason. We are loosing possibly useful clues when programmers include uselessly, or design poor hierarchies for their include files. It now goes a bit far. I was surprised to discover that even Bison adds such protection to the `.h' files it generates, I do not believe there ever was a problem. Python does not exist in such a multiplicity of hierarchies that one is hopeless to ever include properly. So, we could ask ourselves if auto-protection is really meaningful for Python header files... -- François Pinard http://www.iro.umontreal.ca/~pinard From martin@v.loewis.de Fri Oct 11 06:42:11 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 11 Oct 2002 07:42:11 +0200 Subject: [Python-Dev] Multiple Inclusion of Header Files In-Reply-To: <3DA6066A.F349116F@metaslash.com> References: <3DA6066A.F349116F@metaslash.com> Message-ID: Neal Norwitz writes: > is there any reason why these files: > graminit.h > patchlevel.h > pyconfig.h (aka pyconfig.h.in) > > don't protect against being included multiple times? I think they cannot possibly be included multiple times in a correct application, since a correct application does not directly include them, anyway. Regards, Martin From tismer@tismer.com Fri Oct 11 06:46:04 2002 From: tismer@tismer.com (Christian Tismer) Date: Thu, 10 Oct 2002 22:46:04 -0700 Subject: [Python-Dev] Multiple Inclusion of Header Files References: <3DA6066A.F349116F@metaslash.com> <200210102358.g9ANwa811524@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3DA6659C.7040008@tismer.com> Guido van Rossum wrote: ... > Could've been an oversight. I notice that graminit.h and pyconfig.h > are generated files. patchlevel.h was originally a one-liner. > > Is there anything that is gained by adding those? Don't think so, since these are included py Python.h once, and that is protected. -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From ishimoto@gembook.org Fri Oct 11 07:17:11 2002 From: ishimoto@gembook.org (Atsuo Ishimoto) Date: Fri, 11 Oct 2002 15:17:11 +0900 Subject: [Python-Dev] Re: Multibyte repr() In-Reply-To: <200210110004.g9B045011881@pcp02138704pcs.reston01.va.comcast.net> References: <20021011075000.E88A.ISHIMOTO@gembook.org> <200210110004.g9B045011881@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021011135552.474D.ISHIMOTO@gembook.org> Hello, On Thu, 10 Oct 2002 20:04:05 -0400 Guido van Rossum wrote: > Oops. It looks like this is the nail in the coffin of the > locale-specific repr(): a multibyte "character" may end in a backslash > that is not escaped. > > Martin, I'm afraid there's nothing we can do except revert your > multibyte character checkins... > I with to multibyte character checkins remain. Hex coded strings are realy nasty for debugging. As for illegal-pickle issue, it is easy to fix this problem. We just need to provide a simple function that converts string to canonical Python literal representaion, as old repr() does. Regards, -------------------------- Atsuo Ishimoto ishimoto@gembook.org Homepage:http://www.gembook.jp From Paul.Moore@atosorigin.com Fri Oct 11 10:38:29 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Fri, 11 Oct 2002 10:38:29 +0100 Subject: [Python-Dev] Why does the Windows installer use the shortname form of the install directory? Message-ID: <16E1010E4581B049ABC51D4975CEDB8861990E@UKDCX001.uk.int.atosorigin.com> I just installed Python 2.2.2b1 (replacing ActiveState Python) and I = notice that the installer uses the shortname form of the Python install = directory. This appears in sys.path, in file associations (and hence in = sys.argv[0]), and in lots of other places in the registry, not all of = which I follow the reasons for. My Python install directory was C:\Applications\Python - not the = default, but also not containing spaces (which I guess could be a = problem for some scripts). So the short form should not be necessary. = I've fixed the problem manually by hacking the registry, but it's not = something I feel comfortable with... Would it be possible for the installer to use the long filename form (at = least in cases where the install path does not contain spaces)? Sorry I didn't report this earlier, but I've been using the ActiveState = distribution until now, which doesn't have this problem. Paul. From mwh@python.net Fri Oct 11 10:44:45 2002 From: mwh@python.net (Michael Hudson) Date: 11 Oct 2002 10:44:45 +0100 Subject: [Python-Dev] Set-next-statement in Python debuggers In-Reply-To: Richie Hindle's message of "Thu, 10 Oct 2002 21:13:33 +0100" References: <200210091555.g99FtD011873@odiug.zope.com> Message-ID: <2madllxqb6.fsf@starship.python.net> Richie Hindle writes: > Hi, > > [Guido] > > Python should not include features (like writable code object > > attributes) that can cause crashes when used inexpertly. > > If I do implement this, I'll do it in such a way as to make crashes > impossible (my first suggestion said that crashes would be possible, but I > realise that was a mistake). It'll verify that the new value of f_lasti > is a valid bytecode position, and it'll either fix up the stack/blockstack > or refuse to move the position in ways that would affect them. (Armin, I > guess that won't restrict you because you're doing this from C code?) You can still cause crashes though: move f_lasti to point to a STORE_NAME (say) without first executing the corresponding LOAD_CONST (say). Or just execute a LOAD_CONST over and over again until you fall off the end of the stack. > [Armin] > > For safety I'd consider writing the > > frame-object-modifying code in a C extension module, carefully > > documented as "don't use this". > > [Guido] > > That's a reasonable solution: f_lasti should be read-only from Python > > code, but you can write an extension that can write it. > > I'd like to see pure Python debuggers (including pdb) have this feature. > If it's safe to use, might it still be considered for inclusion? (I can't > guarantee I'll have the time to do it, but I'll try.) > > I'm also coming to realise that it's not f_lasti I really want to change, > but f_lineno. The line number is the 'unit' that debuggers use - a > debugger would set f_lineno, and f_lasti would be calculated and changed > to the appropriate value. I don't think that changes any of the arguments > either way, except maybe to distance the idea a little from what Armin is > doing. This is a good point! The scenarios I sketched above would be rather harder (impossible?) to contrive in this situation. Cheers, M. -- 112. Computer Science is embarrassed by the computer. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From mwh@python.net Fri Oct 11 10:57:29 2002 From: mwh@python.net (Michael Hudson) Date: 11 Oct 2002 10:57:29 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects stringobject.c,2.193,2.194 unicodeobject.c,2.171,2.172 In-Reply-To: gvanrossum@users.sourceforge.net's message of "Thu, 10 Oct 2002 17:43:50 -0700" References: Message-ID: <2m7kgpxppy.fsf@starship.python.net> gvanrossum@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Objects > In directory usw-pr-cvs1:/tmp/cvs-serv19831 > > Modified Files: > stringobject.c unicodeobject.c > Log Message: > Fix a nasty endcase reported by Armin Rigo in SF bug 618623: > '%2147483647d' % -123 segfaults. This was because an integer overflow > in a comparison caused the string resize to be skipped. After fixing > the overflow, this could call _PyString_Resize() with a negative size, > so I (1) test for that and raise MemoryError instead; (2) also added a > test for negative newsize to _PyString_Resize(), raising SystemError > as for all bad arguments. > > An identical bug existed in unicodeobject.c, of course. > > Will backport to 2.2.2. No test case? For shame! I'll do one if you don't beat me to it. Cheers, M. -- And then the character-only displays went away (leading to increasingly silly graphical effects and finally to ads on web pages). -- John W. Baxter, comp.lang.python From guido@python.org Fri Oct 11 12:48:52 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 11 Oct 2002 07:48:52 -0400 Subject: [Python-Dev] Multiple Inclusion of Header Files In-Reply-To: Your message of "Fri, 11 Oct 2002 07:42:11 +0200." References: <3DA6066A.F349116F@metaslash.com> Message-ID: <200210111148.g9BBmqu22885@pcp02138704pcs.reston01.va.comcast.net> > > is there any reason why these files: > > graminit.h > > patchlevel.h > > pyconfig.h (aka pyconfig.h.in) > > > > don't protect against being included multiple times? > > I think they cannot possibly be included multiple times in a correct > application, since a correct application does not directly include > them, anyway. But that's true of any .h file that's included by Python.h, isn't it? And there's only a handful of .h files that Python.h doesn't include... --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Fri Oct 11 14:09:16 2002 From: neal@metaslash.com (Neal Norwitz) Date: Fri, 11 Oct 2002 09:09:16 -0400 Subject: [Python-Dev] Multiple Inclusion of Header Files In-Reply-To: <200210111148.g9BBmqu22885@pcp02138704pcs.reston01.va.comcast.net> References: <3DA6066A.F349116F@metaslash.com> <200210111148.g9BBmqu22885@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021011130916.GL30812@epoch.metaslash.com> On Fri, Oct 11, 2002 at 07:48:52AM -0400, Guido van Rossum wrote: > > > is there any reason why these files: > > > graminit.h > > > patchlevel.h > > > pyconfig.h (aka pyconfig.h.in) > > > > > > don't protect against being included multiple times? > > > > I think they cannot possibly be included multiple times in a correct > > application, since a correct application does not directly include > > them, anyway. > > But that's true of any .h file that's included by Python.h, isn't it? > And there's only a handful of .h files that Python.h doesn't include... I don't really think there is much to be gained by adding protection. It's unlikely the files would be included directly by an application. Even if the files are included multiple times, the worst that should happen is a redefined warning. Neal From dave@boost-consulting.com Fri Oct 11 14:26:42 2002 From: dave@boost-consulting.com (David Abrahams) Date: 11 Oct 2002 09:26:42 -0400 Subject: [Python-Dev] Multiple Inclusion of Header Files In-Reply-To: <20021011130916.GL30812@epoch.metaslash.com> References: <3DA6066A.F349116F@metaslash.com> <200210111148.g9BBmqu22885@pcp02138704pcs.reston01.va.comcast.net> <20021011130916.GL30812@epoch.metaslash.com> Message-ID: Neal Norwitz writes: > On Fri, Oct 11, 2002 at 07:48:52AM -0400, Guido van Rossum wrote: > > > > is there any reason why these files: > > > > graminit.h > > > > patchlevel.h > > > > pyconfig.h (aka pyconfig.h.in) > > > > > > > > don't protect against being included multiple times? > > > > > > I think they cannot possibly be included multiple times in a correct > > > application, since a correct application does not directly include > > > them, anyway. > > > > But that's true of any .h file that's included by Python.h, isn't it? > > And there's only a handful of .h files that Python.h doesn't include... > > I don't really think there is much to be gained by adding protection. > It's unlikely the files would be included directly by an application. > Even if the files are included multiple times, the worst that should > happen is a redefined warning. Ahem. Don't forget that C++ applications may need to read these files. The rules about redefinition are different in C++ IIRC. So, if these files /really/ don't need to be #included directy by an application, that's fine... However, I note that Boost.Python has to #include because we have to apply workarounds before #including for certain versions of Python, so I wouldn't say that it's very unrealistic to think applications might need these files separately. -- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From guido@python.org Fri Oct 11 14:40:53 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 11 Oct 2002 09:40:53 -0400 Subject: [Python-Dev] Multiple Inclusion of Header Files In-Reply-To: Your message of "Fri, 11 Oct 2002 09:26:42 EDT." References: <3DA6066A.F349116F@metaslash.com> <200210111148.g9BBmqu22885@pcp02138704pcs.reston01.va.comcast.net> <20021011130916.GL30812@epoch.metaslash.com> Message-ID: <200210111340.g9BDerK23702@pcp02138704pcs.reston01.va.comcast.net> > However, I note that Boost.Python has to #include because > we have to apply workarounds before #including for certain > versions of Python, so I wouldn't say that it's very unrealistic to think > applications might need these files separately. But this works fine without the protection; probably because patchlevel.h contains no C code, only preprocessor stuff. --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Fri Oct 11 14:54:49 2002 From: dave@boost-consulting.com (David Abrahams) Date: 11 Oct 2002 09:54:49 -0400 Subject: [Python-Dev] Multiple Inclusion of Header Files In-Reply-To: <200210111340.g9BDerK23702@pcp02138704pcs.reston01.va.comcast.net> References: <3DA6066A.F349116F@metaslash.com> <200210111148.g9BBmqu22885@pcp02138704pcs.reston01.va.comcast.net> <20021011130916.GL30812@epoch.metaslash.com> <200210111340.g9BDerK23702@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > > However, I note that Boost.Python has to #include because > > we have to apply workarounds before #including for certain > > versions of Python, so I wouldn't say that it's very unrealistic to thi= nk > > applications might need these files separately. >=20 > But this works fine without the protection; probably because > patchlevel.h contains no C code, only preprocessor stuff. Certainly (though PP stuff can cause problems too). I'm just saying that I if you intentionally avoid #include guards on the basis that it's going to help you find your own bugs/inefficiencies, as Fran=E7ois seems to be suggesting, you may be unneccessarily limiting interoperability, which I think is Python's higher goal. --=20 David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com From tim.one@comcast.net Fri Oct 11 16:13:38 2002 From: tim.one@comcast.net (Tim Peters) Date: Fri, 11 Oct 2002 11:13:38 -0400 Subject: [Python-Dev] Multiple Inclusion of Header Files In-Reply-To: Message-ID: Guards in .h files don't hurt, and can help. They're trivial to put in or take out. I vote we ignore "the issue". leading-by-example-ly y'rs - tim From tim.one@comcast.net Fri Oct 11 16:47:30 2002 From: tim.one@comcast.net (Tim Peters) Date: Fri, 11 Oct 2002 11:47:30 -0400 Subject: [Python-Dev] Why does the Windows installer use the shortname form of the install directory? In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8861990E@UKDCX001.uk.int.atosorigin.com> Message-ID: [Moore, Paul] > I just installed Python 2.2.2b1 (replacing ActiveState Python) > and I notice that the installer uses the shortname form of the > Python install directory. This appears in sys.path, in file > associations (and hence in sys.argv[0]), and in lots of other > places in the registry, not all of which I follow the reasons for. > > My Python install directory was C:\Applications\Python - not the > default, but also not containing spaces (which I guess could be a > problem for some scripts). So the short form should not be > necessary. I've fixed the problem manually by hacking the > registry, but it's not something I feel comfortable with... I must have missed what "the problem" is. What is the problem? Path components generally have two distinct names on Windows. That's how the OS works -- we didn't invent this madness . > Would it be possible for the installer to use the long filename > form Yes, but I won't make that change, because we *used* to do it that way and it confused newbies endlessly, mostly because of embedded spaces. By default, the installer suggests an 8.3 name so that nobody can get confused. Pick a non-8.3 name instead, and two distinct names are a fact of Windows Life you have to live with. > (at least in cases where the install path does not contain spaces)? I don't know that the WISE scripting language is capable of making this distinction, and in the absence of an actual problem I'm not going to spend time digging into it. It's certainly too late to make this kind of change in the 2.2 line, so if a volunteer appears who cares enough to address it, it would have to wait for 2.3. From Paul.Moore@atosorigin.com Fri Oct 11 16:59:00 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Fri, 11 Oct 2002 16:59:00 +0100 Subject: [Python-Dev] Why does the Windows installer use the shortname formof the install directory? Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2D0C@UKDCX001.uk.int.atosorigin.com> From: Tim Peters [mailto:tim.one@comcast.net] > [Moore, Paul] > > I just installed Python 2.2.2b1 (replacing ActiveState Python) > > and I notice that the installer uses the shortname form of the > > Python install directory. This appears in sys.path, in file > > associations (and hence in sys.argv[0]), and in lots of other > > places in the registry, not all of which I follow the reasons for. > > > > My Python install directory was C:\Applications\Python - not the > > default, but also not containing spaces (which I guess could be a > > problem for some scripts). So the short form should not be > > necessary. I've fixed the problem manually by hacking the > > registry, but it's not something I feel comfortable with... >=20 > I must have missed what "the problem" is. What is the problem? Path > components generally have two distinct names on Windows. That's how > the OS works -- we didn't invent this madness . "problem" is probably a bit strong. It is user-visible in a few places = (notably sys.path) and is arguably difficult to read in those places. = But it's an issue of aesthetics rather than anything else. OK, so if I have a sense of aesthetics, I probably shouldn't be using = Windows :-) Agreed, it's mainly an OS issue. If Windows didn't do the long/short = pathname idiocy, and if the practice of using pathnames with spaces had = never been promoted, we wouldn't be having this discussion. (Or if Unix = people had got into the habit of actually using the freedom thay have to = use pathnames containing all sorts of strange characters, we may be even = worse off :-)) >=20 > > Would it be possible for the installer to use the long filename > > form >=20 > Yes, but I won't make that change, because we *used* to do it that way = and > it confused newbies endlessly, mostly because of embedded spaces. By > default, the installer suggests an 8.3 name so that nobody can get = confused. > Pick a non-8.3 name instead, and two distinct names are a fact of = Windows > Life you have to live with. OK. If the other way has been shown to cause real problems, then that's = a reason not to do that. Having used ActiveState's distribution until = now (which uses the long form, and advises you not to use paths with = spaces - I've no idea how it copes if you ignore its advice, though...) = I didn't realise that you'd been through this. > > (at least in cases where the install path does not contain spaces)? >=20 > I don't know that the WISE scripting language is capable of making = this > distinction, and in the absence of an actual problem I'm not going to = spend > time digging into it. It's certainly too late to make this kind of = change > in the 2.2 line, so if a volunteer appears who cares enough to address = it, > it would have to wait for 2.3. That's fair enough. And it's not major enough that I'm pushing for = anyone to waste time on it. I just thought I'd mention it in case it was = a simple change to a checkbox, or something. I'm always optimistic on a = Friday - I save the cynicism for Mondays... :-) Thanks for taking the time to comment. Paul. From tim.one@comcast.net Fri Oct 11 17:31:36 2002 From: tim.one@comcast.net (Tim Peters) Date: Fri, 11 Oct 2002 12:31:36 -0400 Subject: [Python-Dev] Why does the Windows installer use the shortname formof the install directory? In-Reply-To: <16E1010E4581B049ABC51D4975CEDB885E2D0C@UKDCX001.uk.int.atosorigin.com> Message-ID: [Moore, Paul] > ... > That's fair enough. And it's not major enough that I'm pushing > for anyone to waste time on it. I just thought I'd mention it in > case it was a simple change to a checkbox, or something. It *almost* is a checkbox! In WISE's idea of "a scripting language", the relevant line looks like this: Set Variable MAINDIR to %MAINDIR% There's not a visible clue that this isn't a no-op. If you double-click on the line in the WISE GUI, though, a dialog box pops up, with a dropdown list labeled "Operation", one of whose entries is Convert to short filename There isn't another variant that adds "but only if it contains a space". Beyond that, I'm stuck -- since Adobe graciously updated my Acrobat Reader, every attempt to view the WISE reference docs (pdf format) hard-freezes (power cycle required) my box. I *like* your suggestion to skip this unless there's an embedded space, I just seem unable to do anything about it now (but it would still be for 2.3 -- mucking with 2.2 is too dangerous). From thomas.heller@ion-tof.com Fri Oct 11 17:47:59 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 11 Oct 2002 18:47:59 +0200 Subject: [Python-Dev] Why does the Windows installer use the shortname formof the install directory? References: Message-ID: <000901c27145$f51e9720$e000a8c0@thomasnotebook> > [Moore, Paul] > > ... > > That's fair enough. And it's not major enough that I'm pushing > > for anyone to waste time on it. I just thought I'd mention it in > > case it was a simple change to a checkbox, or something. [Tim] > > It *almost* is a checkbox! In WISE's idea of "a scripting language", the > relevant line looks like this: > > Set Variable MAINDIR to %MAINDIR% > > There's not a visible clue that this isn't a no-op. If you double-click on > the line in the WISE GUI, though, a dialog box pops up, with a dropdown list > labeled "Operation", one of whose entries is > > Convert to short filename > > There isn't another variant that adds "but only if it contains a space". > > Beyond that, I'm stuck -- since Adobe graciously updated my Acrobat Reader, > every attempt to view the WISE reference docs (pdf format) hard-freezes > (power cycle required) my box. I *like* your suggestion to skip this unless > there's an embedded space, I just seem unable to do anything about it now > (but it would still be for 2.3 -- mucking with 2.2 is too dangerous). > I did a quick check: the online pdf file does say exactly the same as the online help (which is still in windows help format). It pops up when youu press F1 in this dialog. If even this does not work on win 98 , I can post the text here. Thomas From tim.one@comcast.net Fri Oct 11 18:26:08 2002 From: tim.one@comcast.net (Tim Peters) Date: Fri, 11 Oct 2002 13:26:08 -0400 Subject: [Python-Dev] Why does the Windows installer use the shortname formof the install directory? In-Reply-To: <000901c27145$f51e9720$e000a8c0@thomasnotebook> Message-ID: [Tim] > ... > I *like* your suggestion to skip this unless there's an embedded > space, I just seem unable to do anything about it now (but it would > still be for 2.3 -- mucking with 2.2 is too dangerous). [Thomas Heller] > I did a quick check: the online pdf file does say exactly the same > as the online help (which is still in windows help format). > It pops up when youu press F1 in this dialog. If even this does not > work on win 98 , I can post the text here. Better, if this interests you, how about you submit a (tested) patch that skips the convert-to-short business unless there's an embedded space? This isn't going to become a priority for me -- it's a "doctor! doctor!" thing. That is, the installer suggests a path that isn't going to confuse anyone, no way, no how. When someone insists on making life more difficult, my capacity for empathic outrage is limited ; top-level directories haven't been a limited critical resource since FAT faded away. From thomas.heller@ion-tof.com Fri Oct 11 18:31:59 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 11 Oct 2002 19:31:59 +0200 Subject: [Python-Dev] Why does the Windows installer use the shortname formof the install directory? References: Message-ID: <001701c2714c$1aa661c0$e000a8c0@thomasnotebook> > [Tim] > > ... > > I *like* your suggestion to skip this unless there's an embedded > > space, I just seem unable to do anything about it now (but it would > > still be for 2.3 -- mucking with 2.2 is too dangerous). > > [Thomas Heller] > > I did a quick check: the online pdf file does say exactly the same > > as the online help (which is still in windows help format). > > It pops up when youu press F1 in this dialog. If even this does not > > work on win 98 , I can post the text here. > > Better, if this interests you, how about you submit a (tested) patch that > skips the convert-to-short business unless there's an embedded space? This > isn't going to become a priority for me -- it's a "doctor! doctor!" thing. > That is, the installer suggests a path that isn't going to confuse anyone, > no way, no how. When someone insists on making life more difficult, my > capacity for empathic outrage is limited ; top-level directories > haven't been a limited critical resource since FAT faded away. > No, it doesn't interest me. I'm not going to install python (which I often use from the command line) ever into a deep subpath, or one containing spaces. Even my python 1.5.2 lives in c:\Python1.5. I just wanted to remind you on the online (F1) help, which is real good IMO in wise. Thomas From martin@v.loewis.de Fri Oct 11 18:35:52 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 11 Oct 2002 19:35:52 +0200 Subject: [Python-Dev] Multiple Inclusion of Header Files In-Reply-To: References: <3DA6066A.F349116F@metaslash.com> <200210111148.g9BBmqu22885@pcp02138704pcs.reston01.va.comcast.net> <20021011130916.GL30812@epoch.metaslash.com> Message-ID: David Abrahams writes: > However, I note that Boost.Python has to #include because > we have to apply workarounds before #including for certain > versions of Python, so I wouldn't say that it's very unrealistic to think > applications might need these files separately. If people do such things, I think we'ld like to hear about that (atleast if it causes problems). Most likely, people are misguided when they include other Python headers but Python.h (or include Python.h after including system headers); so if there is a problem, it is likely fixed by some other means. If we commit to supporting things we don't know, we might get into troubles some day, having either to break those applications some day, or having to maintain ugly code just because we have to continue to support applications should not have used the feature in the first place. Regards, Martin From kajiyama@grad.sccs.chukyo-u.ac.jp Sat Oct 12 08:24:19 2002 From: kajiyama@grad.sccs.chukyo-u.ac.jp (Tamito KAJIYAMA) Date: Sat, 12 Oct 2002 16:24:19 +0900 Subject: [Python-Dev] Re: Multibyte repr() Message-ID: <200210120724.g9C7OJQ29295@grad.sccs.chukyo-u.ac.jp> Hi, I know it's too late to post this, but I'd like to express my opinion on locale-dependent repr() from a CJKV user's viewpoint. Guido writes: | | I don't think this particular issue (pickling) is a problem. But I | *do* continue to worry that making repr() depend on the locale may be | a bigger problem than what it attempts to solve. I think locale-dependent repr() is a bad idea. The most important (IMHO) aspect of repr() is that returned values of repr() can be used as fragments of Python programs. If repr() were locale-dependent, the returned values could not be used as fragments in some cases. In Japanese applications, Shift_JIS string literals would be a major problem in this sense. I think one of the virtues of Python is that Python has no language feature that is (automagically) affected by locale settings. Applications with locale-specific features would be useful, but I wish Python be as locale-independent as possible. Regards, -- KAJIYAMA, Tamito From tismer@tismer.com Sat Oct 12 12:32:30 2002 From: tismer@tismer.com (Christian Tismer) Date: Sat, 12 Oct 2002 04:32:30 -0700 Subject: [Python-Dev] Why does the Windows installer use the shortname formof the install directory? References: Message-ID: <3DA8084E.3010509@tismer.com> Tim Peters wrote: > [Moore, Paul] > >>... >>That's fair enough. And it's not major enough that I'm pushing >>for anyone to waste time on it. I just thought I'd mention it in >>case it was a simple change to a checkbox, or something. > > > It *almost* is a checkbox! In WISE's idea of "a scripting language", the > relevant line looks like this: > > Set Variable MAINDIR to %MAINDIR% > > There's not a visible clue that this isn't a no-op. If you double-click on > the line in the WISE GUI, though, a dialog box pops up, with a dropdown list > labeled "Operation", one of whose entries is > > Convert to short filename > > There isn't another variant that adds "but only if it contains a space". The F1 key gives you a regular windows help file, as I remember. And you can surround the Convert to short filename thing with an if contruct that checks for a space, something like If MAINDIR Contains " " then # here the Convert thingie End ciao - chris p.s.: The whol script is in ASCII. I used to generate scripts from WISE templates which I've written once. -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo@ulb.ac.be Sat Oct 12 18:45:02 2002 From: arigo@ulb.ac.be (Armin Rigo) Date: Sat, 12 Oct 2002 19:45:02 +0200 Subject: [Python-Dev] Memory size overflows Message-ID: <3DA85F9E.339DAC1F@ulb.ac.be> Hello everybody, All around the C code there are potential problems with objects of very large sizes (http://www.python.org/sf/618623). The problem is that to allocate a variable-sized object of type 't' with 'n' elements we compute 'n*t->tp_itemsize', which can overflow even if 'n' is a perfectly legal value. If the truncated result is small, the subsequent malloc() suceeds, and we run into a segfault by trying to access more memory than reserved. The same problem exists at other places -- more or less everywhere we add or multiply something to a number that could be user-supplied. For example, Guido just fixed '%2147483647d'%-123. A rather artificial example, I agree, but a hole anyway. To fix this I suggest introducing a few new macros in pymem.h that compute things about sizes with overflow checking. I can see a couple of approaches based on special values that mean "overflow": 1) there is just one special "overflow" value, e.g. ((size_t)-1), that is returned and propagated by the macros when an overflow is detected. This might be error-prone because if we forget once to use the macros to add a few bytes to the size, this special value will wrap down to a small legal value -- and segfault. 2) same as above, but with a whole range of overflow values. For example, just assume (or decide) that no malloc of more than half the maximum number that fits into a size_t can succeed. We don't need any macro to add a (resonable) constant to a size. We need a macro for multiplication that -- upon overflow -- returns the first number of the "overflow" range. The Add macro is still needed to sum *two* potentially large numbers. 3) we compute all sizes with signed integers (int or long), as is currently (erroneously?) done at many places. Any negative integer is regarded as "overflow", but the multiplication macro returns the largest negative integer in case of overflow, so that as above no addition macro is needed for the simple cases. This will require a "multiplication hunt party" :-) Also, approaches 2 and 3 require fixes to ensure that 'malloc(any-overflow-size)' always fail, for any of the several implementations of malloc found in the code. Even with approach 1, I would not trust the platform malloc to correctly fail on malloc(-1) -- I guess it might "round up" the value to 0 before it proceed... Armin From tim.one@comcast.net Sun Oct 13 01:06:37 2002 From: tim.one@comcast.net (Tim Peters) Date: Sat, 12 Oct 2002 20:06:37 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: <3DA85F9E.339DAC1F@ulb.ac.be> Message-ID: [Armin Rigo] > All around the C code there are potential problems with objects of very > large sizes (http://www.python.org/sf/618623). The problem is that to > allocate a variable-sized object of type 't' with 'n' elements we > compute 'n*t->tp_itemsize', which can overflow even if 'n' is a > perfectly legal value. I confess I always ignore these until one pops up in real life. Checking slows the code, and that causes me pain <0.5 wink>. Note that the multiplication isn't the only problem: we usually go on to add in the size of the object header, and that can overflow too. This leads to heartbreaking code such as string_repeat's: nbytes = size * sizeof(char); if (nbytes / sizeof(char) != (size_t)size || nbytes + sizeof(PyStringObject) <= nbytes) { PyErr_SetString(PyExc_OverflowError, "repeated string is too long"); return NULL; } Making it worse, while malloc takes an unsigned arg, Python usually uses a signed type to hold these things. When I fixed the code above (it did pop up in real life there), I made sure nbytes was declared size_t, so that the compiler could do the division-by-constant with a simple shift (well, OK, sizeof(char) is necessarily 1U, so the division is pointless there anyway; in other contexts it's not, and signed/power_of_2 is more expensive than unsigned/power_of_2 because most C compilers don't allow the former to compute the natural floor; note that anything/non_constant can be incredibly expensive!). > If the truncated result is small, the subsequent malloc() suceeds, and > we run into a segfault by trying to access more memory than reserved. > The same problem exists at other places -- more or less everywhere we > add or multiply something to a number that could be user-supplied. Yup -- and that, in a nutshell, is why I usually ignore these . Note that it's not enough even to ensure malloc gets a sensible argument: on at least two flavors of Unix, malloc() returning a non-NULL pointer doesn't guarantee you can access the memory without segfaulting. In the end, it's a game that can't be won. It may be nicer to do a little better, but there are real costs too. > ... > To fix this I suggest introducing a few new macros in pymem.h that > compute things about sizes with overflow checking. I can see a couple > of approaches based on special values that mean "overflow": If we're going to this, I suggest an ubiquitous trick from Zope's C code: DO_SOMETHING_OR(RESULT_LVALUE, INPUT1, ..., ON_ERROR_BLOCK); That is, the result goes into RESULT_LVALUE, unless there's an overflow; in the latter case, ON_ERROR_BLOCK is executed. This is usually something like "return NULL;" or "goto Overflow;". This has the nice property of pushing the error-case code out of the normal flow. A goto is especially nice, because the label provides a handy place to set a debugger breakpoint (always a practical problem when code hides in macro expansions). > ... > 3) we compute all sizes with signed integers (int or long), > as is currently (erroneously?) done at many places. Any > negative integer is regarded as "overflow", but the > multiplication macro returns the largest negative integer in > case of overflow, so that as above no addition macro is > needed for the simple cases. There's really never been anything special about "the sign bit", and some 32-bit boxes can allocate 1UL<<31 bytes. This becomes more of a problem as Python gets applied to larger problems, and 2 gigabytes is no longer an insane amount of address space. > ... > Also, approaches 2 and 3 require fixes to ensure that > 'malloc(any-overflow-size)' always fail, for any of the several > implementations of malloc found in the code. pymalloc is already suitably paranoid, even to the extent that the debug pymalloc makes sure adding 16 byes for its padding doesn't overflow. > Even with approach 1, I would not trust the platform malloc to > correctly fail on malloc(-1) -- I guess it might "round up" the > value to 0 before it proceed... malloc() can't get an argument of -1: it takes a size_t argument, and size_t is guaranteed to be unsigned. What *does* happen is that some platform mallocs don't do what the debug pymalloc does, meaning that they don't check that adding in their own overhead doesn't overflow. So, e.g., if they need a 4-byte header for bookkeeping, they'll add 4 to the argument and not even notice if that "wraps around" to 0, 1, 2, or 3. Disaster ensues. But it's not really Python's job to fix broken platform libraries. lukewarmly y'rs - tim From tismer@tismer.com Sun Oct 13 06:37:43 2002 From: tismer@tismer.com (Christian Tismer) Date: Sat, 12 Oct 2002 22:37:43 -0700 Subject: [Python-Dev] Memory size overflows References: Message-ID: <3DA906A7.3030408@tismer.com> Tim Peters wrote: > [Armin Rigo] > >>All around the C code there are potential problems with objects of very >>large sizes (http://www.python.org/sf/618623). The problem is that to >>allocate a variable-sized object of type 't' with 'n' elements we >>compute 'n*t->tp_itemsize', which can overflow even if 'n' is a >>perfectly legal value. > > > I confess I always ignore these until one pops up in real life. Checking > slows the code, and that causes me pain <0.5 wink>. > > Note that the multiplication isn't the only problem: we usually go on to > add in the size of the object header, and that can overflow too. This leads > to heartbreaking code such as string_repeat's: (sigh) [snipped all the rest of good stuff] May I throw in a "weird" idea, again? I'm thinking since quite some time that Python should really abandon 32 bit arithmetic once and forwver and switch to use 64 bits, everywhere. It would not be the first scripting language which does so (wasn't it Ruby?), and it would benefit from that quite much, I think. I believe that 32 bit technology is outworn like 16 bits have before. When? Well, I'd say 16 years ago, exactly, if we assume an increase of memory size of 1 bit per year, which seems realistic. There are already a couple of 64 bit machines running Python. These would not see any slow-down by switching to 64 bits. New upcoming processors a likely to be 64 bitters pretty soon, and I hope to see such a nice machine like Knuth's MMIX in the next five years or earlier. Most probably, 64 bits will then be state of the art, and by switching to it right now, we are solving a lot of problems for the next 32 years ahead. This is beyond the careers of most of the "old" guys around, giving us a chance to solve some problem "once and forever". The current small speed loss will vanish from alone :) It-might-sound-funny-but-it-*is*-my-opinion - ly y'rs chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From drifty@bigfoot.com Sun Oct 13 09:03:08 2002 From: drifty@bigfoot.com (Brett Cannon) Date: Sun, 13 Oct 2002 01:03:08 -0700 (PDT) Subject: [Python-Dev] Python-dev Summary for 2002-10-01 through 2002-10-13 Message-ID: I realize this is a little early, but I am going on vacation on Monday for two weeks (Raymond Hettinger has agreed to do the summary while I am gone), so I wanted to get this out the door before I leave. Usual 24 hour correction period for the list to make my writing skills and comprehension skills seem inferior is in effect. =) -------------------------------------------------------------------- This is a summary of traffic on the `python-dev mailing list`_ between October 1, 2002 and October 13, 2002 (inclusive). It is intended to inform the wider Python community of on-going developments on the list that might interest the wider Python community. To comment on anything mentioned here, just post to python-list@python.org or comp.lang.python in the usual way; give your posting a meaningful subject line, and if it's about a PEP, include the PEP number (e.g. Subject: PEP 201 - Lockstep iteration). All python-dev members are interested in seeing ideas discussed by the community, so don't hesitate to take a stance on a PEP (or anything else for that matter) if you have an opinion. And if all of this really interests you then get involved and join Python-dev! This is the fourth summary written by Brett Cannon (with a partially fried brain thanks to the GRE). Summaries by me (2002-09-15 to ... when I burn out) are archived at: http://www.ocf.berkeley.edu/~bac/python-dev/summaries/index.php You can find summaries by Michael Hudson (2002-02-01 to 2001-07-05) at: http://starship.python.net/crew/mwh/summaries/index.html Summaries by A.M. Kuchling (2000-12-01 to 2001-01-31) are at: http://www.amk.ca/python/dev/ Please note that this summary is written using reStructuredText_ which can be found at http://docutils.sourceforge.net/rst.html . Any unfamiliar punctuation is probably markup for reST; you can safely ignore it (although I suggest learning reST; its nice and is accepted for PEP markup). Also, because of the wonders of reformatting thanks to whatever you are using to read this, I cannot guarantee you will be able to run this text through DocUtils as-is. If you want to do that, get the original text from the archive. .. _python-dev mailing list: http://mail.python.org/mailman/listinfo/python-dev .. _reST: .. _reStructuredText: http://docutils.sf.net/ ====================== Summary Announcements ====================== This is a new section to the summary that I have decided to introduce. It is mainly going to serve to make any general announcements or comments on this summary and this summary alone. All universal comments will stay at the top of the summaries. Just to let everyone know, I am taking off for two weeks on vacation starting 2002-10-14 and I will not return until 2002-10-30. Now, before you all start sobbing over the loss of one of my great summaries, you should know that Raymond Hettinger has graciously taken up the job of temp summarizer for me and will do the summary while I am gone. Michael Hudson has made the suggestion that I inject more of my personality into the summary so as to liven it up a little. I am personally quite happy to do this. But the real question is do you, fine reader, mind the idea? If I don't hear from throngs of people going "your sarcastic tone takes away from the wonderfully drawl summaries and that is a bad thing", then I will just go ahead and write with personality. Just don't complain later. =) 2.2.2b1 has been occupying Python-dev during this summary period, and so this summary is shorter than usual. I left out a bunch of threads that were discussing bugfixes that I either didn't find interesting or didn't think the rest of the world would care about. Now it's time for my personal favor of the month. I am going to start applying to grad school for computer (science | programming) when I get back from vacation. If anyone knows of Python-friendly schools out there, let me know. Heck, I am even willing to leave America to go to school as long as the classes are in English. So if you know of any, please let me know! And now on to the summary. ===================================== `Python 2.2.2 beta release on Monday!`__ ===================================== __ http://mail.python.org/pipermail/python-dev/2002-October/029204.html splintered thread: RELEASED: Python 2.2.2b1 : http://mail.python.org/pipermail/python-dev/2002-October/029337.html Python 2.2.2.b1 was released on Monday, October 7. This is the reason (or perhaps excuse is a better description) for the lighter summary this week. A good amount of the traffic on Python-dev was about bugfixing 2.2.2.b1. Most of this probably would not interest the average Python user, and thus I didn't summarize a bunch of threads. Taking the GRE also didn't help with my free time and thus has caused me to cut down on the summary since I am having to go through a huge backlog to get this out the door. ===================================== `Dropping support for Tcl 8.0 and 8.1`__ ===================================== __ http://mail.python.org/pipermail/python-dev/2002-October/029024.html Martin v. Loewis asked if it would be okay to drop support for Tcl_ 8.0 and 8.1 since `_tkinter.c`_ has special code in there just for those outdated versions. Guido ok'ed it, so if you are using those still using a version of Tk from way back when, it's time to upgrade. .. __tkinter.c: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Modules/_tkinter.c ======================================= `*very* revised proposal for interfaces`__ ======================================= __ http://mail.python.org/pipermail/python-dev/2002-October/029042.html Previously John Williams came up with a proprosal for implementing the stuff from `PEP 245`_ (Python Interface Syntax) and `PEP 246`_ (Object Adaptation). From my understanding of what John has done, it appears he has written an interface system in pure Python. If you want backstory to this long and involved discussion to get an interface system into Python read the Python-dev Summaries for 2002-08-16 to 2002-09-01 and 2002-09-01 to 2002-09-15 . Gerald Williams, Michael Chermside, and Esteban Castro all commented on the implementation and made various suggestions. John said that he was not done yet implementing this. But if you were interested in the whole previous discussion on interfaces you could consider looking at what John has done. As for it going into the language, I suspect that will have to wait until John is done and has convinced the PEP writers and Python-dev that his implementation fits the bill. Stay tuned. If you want some more background info on interfaces, read the previously mentioned summaries. As for object adaptation, read on c.l.py and on Python-dev anything by Alex Martelli on the subject. He has become the main proponent of object adaptation and has written several very extensive essays on the subject. .. _PEP 245: http://www.python.org/peps/pep-0245.html .. _PEP 246: http://www.python.org/peps/pep-0246.html ================= `perplexed by mro`__ ================= __ http://mail.python.org/pipermail/python-dev/2002-October/029035.html Splinter threads: Re: my proposals about mros (was: perplexed by mro) : http://mail.python.org/pipermail/python-dev/2002-October/029167.html C3 implementation : http://mail.python.org/pipermail/python-dev/2002-October/029230.html Samuele Pedroni said he was "trying to wrap [his] head around the mro computation in 2.2". Apparently there is the algorithm mentioned at http://www.python.org/2.2.1/descrintro.html (dubbed the naive algorithm) and then the one implemented in typeobject.c_ (called the 2.2 algorithm). Samuele discovered some inconsistencies with the implemented algorithm that he desired some explanation about. Guido responded, thankful that someone was giving this a look because his "intuition about the equivalence between algorithms turned out to be wrong". Guido stated that he thought that he wrote the algorithm from the book "Putting Metaclasses To Work" correctly sans raising an error when major conflicts occur in the ordering. In a later email Guido explained that the naive algorithm came about by his attempt to simplify the explanation of the 2.2 algorithm. Guido pretty much wrote the algorithm from the aforementioned book. Now the algorithm is not simple, so Guido did his best to simplify the explanation. Unknowningly, though, he came up with a variant on the algorithm in his explanation. Greg Ewing pointed out that he thought the naive algorithm was nicer since it seemed to work more intuitively and was easier to explain (and remember kids, these are basic tenants in Python programming). Guido ended up stating that "If Samuele agrees that the naive algorithm works better, [Guido will] try to make it so in 2.3". Well, Samuele said that the "2.2 mro is the worst of our options". There was a problem, though, with the naive algorithm; it is not monotonic as pointed out by Samuele. This led him to put out two options: 1. Use the naive algorithm, which had the drawback of being monotonic. Samuele also believed that it didn't produce "the most natural results". 2. Adopt C3_ as described at http://www.webcom.com/haahr/dylan/linearization-oopsla96.html and apparently used by Dylan_ and Goo_ . This algorithm is monotonic and Samuele says is more intuitive. Guido got around to reading the C3_ paper and agreed that "we should adopt C3". He thought that the 2.2 algorithm was like the C*[LOOPS] algorithm mentioned in the paper. Samuele then wrote a C implementation of the algorithm. Guido said he would get to the patch after 2.2.2b1 got out the door. .. _C3: http://www.webcom.com/haahr/dylan/linearization-oopsla96.html .. _Dylan: http://monday.sourceforge.net/wiki/ .. _Goo: http://www.ai.mit.edu/~jrb/goo/manual.43/goomanual_55.html .._typeobject.c: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Objects/typeobject.c =========================== `Psyco requests and patches`__ =========================== __ http://mail.python.org/pipermail/python-dev/2002-October/029038.html Armin Rigo of psyco_ an email to Python-dev with some thoughts on psyco and some requests. After mentioning how he wished more people realized psyco is meant to be used by anyone and not speed-hungry coders (it is a cool app and if you have any interest in compilers you should take a look) and that psyco could get some more advertising he mentioned three patches that he wrote (patches 617309_ , 617311_ , and 617312_ ) against 2.2.2 that he would like to see be accepted so as to ease maintenance of psyco. Armin also mentioned how he would like to move psyco forward. He pointed out he would like to eventually write it all in Python. This would require tracking changes in the interpreter that psyco dealt with. He would like to keep this in mind when Python 3 discussions kick up (and don't ask when that will happen; not for a VERY long time). In regards to Armin's patches, Martin v. Loewis thought they broke binary compatibility (big no-no between micro releases), but Armin claimed it didn't. After glancing over the patches it seems they have all been applied against 2.2.2 and are being actively worked on by Armin for application to 2.3. .. _617309: http://www.python.org/sf/617309 .. _617311: http://www.python.org/sf/617311 .. _617312: http://www.python.org/sf/617312 .. _psyco: http://psyco.sf.net/ =================================================================== `PEP239 (Rational Numbers) Reference Implementation and new issues`__ =================================================================== __http://mail.python.org/pipermail/python-dev/2002-October/029065.html splintered threads: Re: PEP239 (Rational Numbers) Reference Implementation and new issues : http://mail.python.org/pipermail/python-dev/2002-October/029068.html Christopher Craig uploaded `patch 617779`_ that implemented `PEP 239`_ (Rational Numbers). He had some questions for Python-dev, though, regarding a couple points. These few points have become the bane of my summarizing existence; this thread is huge. One was whether division should return rationals instead of floats. Since rationals keep precision in division they are the most accurate way to perform division. THey also make the most output sense (e.g. 1/3). Problem with this is that rational math is slow and this would cause issue with any code that expected a float. The next issue was about comparison. Should a rational compare only when it is exactly equal to a float or when the float is really close? Lastly, Christopher wondered if rationals should hash the same as floats. The answer to the second issue would influence the answer to this issue. Issue 1. Eric Raymond was for returning a rational. Francois was kind of on the fence. Christian was +1 for returning rationals. Guido said ABC did this and that numeric processes thus ended up being slow. Issue 2. Christian said "Let it grow! Let the user feel what precision he's carrying around, and how much they throw away when they reduce down to a float." Issue 3. Eric Raymond suggested a global "fuzz" variable that defines a "close-enough-for-equality range"; this idea was used by APL. Andrew Koenig was against this because you don't always want a fuzzy comparison and it destroys substitutability: "If a==b, it is not always true that f(a)==f(b)". Andrew said he preferred Scheme's numeric model. To this, Guido said that "'It works in Scheme' doesn't give me a warm fuzzy feeling that it's been tried in real life"; Tim later laid the smack down on Scheme's numeric model and ended it with "There's a reason the NumPy folks never bug you for Scheme features ". Christian pointed out that keeping it as a rat would prevent overflows from ever occuring from long division. Tim was staunchly against a fuzz variable. Raymond suggested a fuzz comparison function that took in a fuzz value. Christopher said that the way it stands now in the implementation is that rationals are coerced to floats and then compared. Oren Tirosh suggested a thired boolean, 'Undetermined', that would be raised when the "difference between A and B is below the error margin". David Abrahams said that Boost discussed this and said that the cost of adding ternary logic was not worth it. Andrew asked how rats could be optimized. He suggested ditching trailing zeros. Tim wondered how much of a save you would get from this. Raymond Hettinger suggested having a builtin variable that would specify the "maximum denominator magnitude". Christian liked this idea. Greg didn't think this would be a good solution because people using rats are going to want them specifically because they are exact. So Raymond suggest the default be unrestricted denominator. Guido brought up the question of how rats should be represented when printed. The syntax for rats came up. Greg Ewing got the ball rolling by suggesting the syntax for rat division as ``\\\``. M.A. Lemburg suggested just having a constructor like ``rat(2,3)``. The discussion then had a gamut of suggested syntax: ``2:3``, ``2r3``, ``{2/3}`` (Guido shot this down because he wants to leave the option open for possible set notation), ``<2/3>``,``2r/3``, something by Barry using an extended character that Pine wouldn't display (it was a joke), and finally ``2/3r`` by Guido. People agreed that this last one suggested by Guido was the best one. Tim also pointed out that Scheme has notation to specify whether a number is exact or not and using the 'r' notation would basically provide the same functionality. But regardless of what syntax people preferred, it was overwhelmingly agreed that choosing the syntax should wait until rationals have been in the language for a while and it is known how they are used. If you only read one thing, read Tim's emails since he explains all of this really well and is the resident math whiz on Python-dev. .. _patch 617779: http://www.python.org/sf/617779 .. _PEP 239: http://www.python.org/peps/pep-0239.html ============================================== `Non-ASCII characters in test_pep277.py in 2.3`__ ============================================== __ http://mail.python.org/pipermail/python-dev/2002-October/029247.html Guido pointed out that test_pep277.py_ uses an encoding cookie which was not being recognized by his toolchain. At some point he stated that he is "still not 100% comfortable with using arbitrary coding cookies in the Python distribution". The reason I mention this thread (beyond for the quote above) is that info on how to get XEmacs to recognize the cookie came out. Sjoerd Mullender sent out the link http://www.xemacs.org/Documentation/packages/html/mule-ucs_2.html which helped some people. As for Linux distro-specific problems, M.A. Lemburg noticed that SuSE puts Mule support in a package named 'mule-ucs-xemacs' and once he got the package loaded XEmacs worked. .. _test_pep277.py: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Lib/test/test_pep277.py ================================================== `Unclear on the way forward with unsigned integers`__ ================================================== __http://mail.python.org/pipermail/python-dev/2002-October/029249.html Splinter threads: PEP 263 in the works : http://mail.python.org/pipermail/python-dev/2002-October/029289.html Mark Hammond was "a little confused by the new world order for working with integers in extension modules". Mark wanted know how to create objects that were more like a collection of bits than an integer. Tim suggested creating a Python long; that would act more like an unsigned int in terms of its bits. The FutureWarning for hexadecimal constants was brought up and it was pointed out that to deal with those just stick an 'L' at the end. Remember folks, that in Python 2.3 ``0x80000000L == 2147483648``. The usefulness of __future__ statements also came up. Tim wondered how useful they were. Thomas Wouters, though, came to __future__'s defense and explained how it helped him migrate people to newer versions of Python without being yelled at for breaking their code. ============================================= `segmentation fault with python2.3a0 from cvs`__ ============================================= __ http://mail.python.org/pipermail/python-dev/2002-October/029286.html Gersen Kurz was having problems importing a huge file. The bug was attributed to cygwin's malloc implementation, so people might want to watch out for that. It was also pointed out that a loop with a bunch of items in a dict created a huge number of references. It turns out that dicts use dummy references in its implementation for when something is deleted. So don't be alarmed by huge references even after you deleting an immense dict. ====================================== `Snapshot win32all builds of interest?`__ ====================================== __ http://mail.python.org/pipermail/python-dev/2002-October/029352.html As mentioned in the last summary, Mark Hammond wondered if anyone would care to have access to compiled snapshots of CVS for Windows. He got enough of a response to give access at http://starship.python.net/crew/mhammond . This is not like the standard Windows installer, though; "This version installs no shortcuts, does not compile .pyc files etc - you are pretty much on your own. Pythonwin\start_pythonwin.pyw is installed to start Pythonwin, but you must do so manually". Mark would like to know if you end up using this. ======================================= `Set-next-statement in Python debuggers`__ ======================================= __ http://mail.python.org/pipermail/python-dev/2002-October/029377.html Richie Hindle wants to write a pure Python debugger; the problem is that it would be difficult without certain C-level stuff exposed to Python. Specifically, he wants frame.f_lasti (to be found in frameobject.c_ ) to be writable by Python so he "could implement Set-Next-Statement". Michael Hudson is the first email I have of someone chiming against this. He thought it would be better to make it a descriptor so as that "you can do some sanity checking on the values". Guido then chimed in saying that if it was writable that would open up a hole for crashing a program. Guido eventually said making it read-only would be fine. This led to Armin Rigo pointing out that you can crash the interpreter already with the new module "or by writing crappy .pyc files". Guido acknowledged this, but said he didn't want to add anymore if it could be helped. He pointed out that he wants to stick with the idea that a segfault is Python's fault unless proven otherwise. .. _frameobject.c: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Objects/frameobject.c ================= `Multibyte repr()`__ ================= __ http://mail.python.org/pipermail/python-dev/2002-October/029442.html A patch was applied that allowed repr() to return characters with the high bit set; repr() used the "multibyte C library for printing string if available". This had caused a bug and made Guido wonder if this was a good thing to do. For an example:: >>> u = u'\u1f40' # Python 2.2 >>> s = u.encode('utf8') >>> s '\xe1\xbd\x80' >>> > > > u = u ' \ u 1 f 4 0 ' > > > s = u . e n c o d e ( ' u t f 8 ' ) > > > s '12 \ x 8 0 ' # Notice the extended character > > > "The latter output is not helpful, because the encoding of s is not the locale's encoding". Martin v. Loewis said that he thought author of the patch's intention was "to get 'proper' output in interactive mode for strings". Part of the issue with all of this is the GNU readline calls setlocale() automatically. A patch came about to reset it in the extension module back to its original state. The issue of whether pickling would break because of this. Guido tried it and had no issue. Atsuo Ishimoto (who brought up the possible problem) said it broke when using the ShiftJFS locale. But the worry of having repr() be locale-specific still lingered. Martin said he was "convinced that having repr locale-specific is unacceptable". He said, though, that having the tp_print slot use a locale-aware print function was fine and to have it differ from tp_repr was fine. But this was shot down by Guido; "tp_print only gets invoked when sys.stdout is a real file; otherwise str() or repr() get invoked". Apparently tp_print is a performance optimization and thus should be fully transparent and not be different in any way to the user. Guido said that the multibyte-string patch should be backed out. With the pickle issue and different semantics for sys.stdout because of tp_print, Guido said the patch had to be backed out. As Tamito Kajiyama said, "one of the virtues of Python is that Python has no language feature that is (automagically) affected by locale settings". =================================== `tp_dictoffset calculation in 2.2.2`__ =================================== __ http://mail.python.org/pipermail/python-dev/2002-October/029502.html Guido asked David Abrahams and Kevin Jacobs if a change in how tp_dictoffset (found in typeobject.c_ ) was calculated would affect them. To give some background, tp_dictoffset "tells us where the instance dict pointer is in the instance object layout". This was a three step process (all Guido's words): 1. (Line 1113) For dynamically created types (i.e. created by class statements), if the new type has no __slots__, the dominant base class has a zero tp_dictoffset, and the dominant base doesn't have a custom tp_getattro, a __dict__ will be added to the instance layout, and its tp_dictoffset is calculated by taking the end of the base class instance struct (or in a more complicated way if tp_itemsize is nonzero). 2. (Line 1941) If the dominant base has a nonzero tp_dictoffset, copy the tp_dictoffset from the dominant base. 3. (Line 2090) The tp_dictoffset of the first non-dominant base class that has a nonzero tp_dictoffset is copied. That last rule had caused Guido and Jeremey Hylton some problems with some code they were bugfixing. Guido wanted to just get rid of that rule since he though "it is *always* wrong". Both David and Kevin said that nothing broke for them, so that is now all straightened out. ====================== `Memory size overflows`__ ====================== __ http://mail.python.org/pipermail/python-dev/2002-October/029535.html Armin Rigo pointed out some potential overflow errors "with objects of very large sizes". The issue was when the amount of memory needed to allocate was calculated there was the chance that it would overflow. Armin suggested adding macros to deal with various issues. Arrive Tim Peters, creator of pymalloc. He admitted that he "always ignore[s] these [errors] until one pops up in real life" since "Checking slows the code, and that causes [Tim] pain <0.5 wink>". He pointed out another possible overflow calculation problem. But it was a basically a hopeless battle since malloc() has its own cross-platform issues. But Tim did say that if it was decided to go down the macro route, then he wanted something like what Zope does: ``DO_SOMETHING_OR(RESULT_LVALUE, INPUT1, ..., ON_ERROR_BLOCK);``. The result goes into RESULT_LVALUE unless there is a problem, in which case ON_ERROR_BLOCK is run. Christian Tismer chimed in and said that he thought we should just move completely over to 64 bit math. Ruby had done it successfully so it wasn't like we were taking a blind leap. It would also save us the hassle from doing it down the road when 64 bit processors become the norm instead of the exception. From oren-py-d@hishome.net Sun Oct 13 12:21:36 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Sun, 13 Oct 2002 07:21:36 -0400 Subject: [Python-Dev] Why does the Windows installer use the shortname formof the install directory? In-Reply-To: <3DA8084E.3010509@tismer.com> References: <3DA8084E.3010509@tismer.com> Message-ID: <20021013112136.GA77207@hishome.net> On Sat, Oct 12, 2002 at 04:32:30AM -0700, Christian Tismer wrote: > >There isn't another variant that adds "but only if it contains a space". > > The F1 key gives you a regular windows help file, as I > remember. > And you can surround the > Convert to short filename > thing with an if contruct that checks for a space, something like > If MAINDIR Contains " " then > # here the Convert thingie > End Translated to WISEspeak: Parse String "%MAINDIR%" into BEFORESPACE and AFTERSPACE [*] If AFTERSPACE Not Equal "" then Set Variable MAINDIR to %MAINDIR% End [*] Set "Pattern/Position" field to single space (invisible in dialog box) Set "Operation" field to "Split value at first occurence of pattern" Oren From skip@manatee.mojam.com Sun Oct 13 13:00:18 2002 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 13 Oct 2002 07:00:18 -0500 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200210131200.g9DC0InD000870@manatee.mojam.com> Bug/Patch Summary ----------------- 310 open / 2943 total bugs (+15) 96 open / 1727 total patches (-16) New Bugs -------- PythonLauncher Dialogs do not work (2002-10-06) http://python.org/sf/619245 test_builtin hangs on MacOSX 10.2.1 (2002-10-06) http://python.org/sf/619400 Import fails in Idle (2002-10-07) http://python.org/sf/619713 imaplib fails with literals in LIST resp (2002-10-07) http://python.org/sf/619732 segmentation fault importing huge source (2002-10-07) http://python.org/sf/619789 __ipow__ broken for 'object's (2002-10-08) http://python.org/sf/620179 inspect and object instances (2002-10-08) http://python.org/sf/620190 HTMLParser:endtag events in comments (2002-10-08) http://python.org/sf/620243 core setup.py fails on Windows (2002-10-08) http://python.org/sf/620364 Max recursion limit with "*?" pattern (2002-10-08) http://python.org/sf/620412 distutils mixed stdin/stdout output (2002-10-08) http://python.org/sf/620630 websucker relative-URL errors (2002-10-09) http://python.org/sf/620705 missing mappings in locale tables (2002-10-09) http://python.org/sf/620739 SocketServer/socket allow_reuse_address (2002-10-09) http://python.org/sf/620783 File write examples are inadequate (2002-10-09) http://python.org/sf/621057 email: ASCII decoding error on 8bit data (2002-10-10) http://python.org/sf/621457 Tix Checklist getselection getstatus bug (2002-10-10) http://python.org/sf/621511 sizeof(struct stat) changes (2002-10-10) http://python.org/sf/621523 docs do not include spec info from PEPs (2002-10-10) http://python.org/sf/621526 Numerous defunct threads left behind (2002-10-10) http://python.org/sf/621548 menues in emacs21 in transient mode (2002-10-10) http://python.org/sf/621554 httplib HEAD request fails - keepalive (2002-10-11) http://python.org/sf/622042 New Patches ----------- C3 MRO algorithm implementation (2002-10-06) http://python.org/sf/619475 tkFileDialog multiple option (2002-10-11) http://python.org/sf/621891 dummy_thread.py implementation (2002-10-13) http://python.org/sf/622537 Closed Bugs ----------- header file problems (2002-09-10) http://python.org/sf/607253 Numpy doesn't build for Python.framework (2002-09-17) http://python.org/sf/610730 broken link in documentation (2002-09-25) http://python.org/sf/614821 Connecting to IPv4 addrs from IPv6 socke (2002-10-04) http://python.org/sf/618594 list(xrange(sys.maxint/4)) again (2002-10-04) http://python.org/sf/618623 Closed Patches -------------- a move function for shutil.py (2001-08-04) http://python.org/sf/448038 --python-build for install (2001-09-05) http://python.org/sf/458898 Multibyte string on string::string_print (2001-11-08) http://python.org/sf/479898 binary packagers (2002-03-19) http://python.org/sf/531901 2.2 patches for BSD/OS 5.0 (2002-03-26) http://python.org/sf/535335 Webbrowser.py and konqueror (2002-04-04) http://python.org/sf/539360 addition of cmath.arg function (2002-04-16) http://python.org/sf/544909 Fix broken smtplib.login() (2002-05-05) http://python.org/sf/552605 Mutable object change flag (2002-05-12) http://python.org/sf/555251 Implementation of major, minor and makedev (2002-06-14) http://python.org/sf/569139 AUTH method LOGIN for smtplib (2002-06-21) http://python.org/sf/572031 Adds Galeon support to webbrowser.py (2002-07-24) http://python.org/sf/585913 galeon support in webbrowser (2002-07-25) http://python.org/sf/586437 Bugfix for urllib2.py (2002-08-25) http://python.org/sf/599836 Updated .spec file for 2.2 series. (2002-09-18) http://python.org/sf/611191 "Bare" text tag_configure in Tkinter (2002-09-21) http://python.org/sf/612602 fix for urllib2.AbstractBasicAuthHandler (2002-09-25) http://python.org/sf/614596 textwrap: prefix short lines (2002-09-30) http://python.org/sf/616496 Fix test_pep277 on Win9x (2002-10-05) http://python.org/sf/619108 From pedronis@bluewin.ch Sun Oct 13 16:22:19 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 13 Oct 2002 17:22:19 +0200 Subject: [Python-Dev] Python-dev Summary for 2002-10-01 through 2002-10-13 References: Message-ID: <008e01c272cc$521e2f80$6d94fea9@newmexico> From: "Brett Cannon" > ================= > `perplexed by mro`__ > ================= > __ http://mail.python.org/pipermail/python-dev/2002-October/029035.html > > Splinter threads: > > Re: my proposals about mros (was: perplexed by mro) : > http://mail.python.org/pipermail/python-dev/2002-October/029167.html > C3 implementation : > http://mail.python.org/pipermail/python-dev/2002-October/029230.html > > Samuele Pedroni said he was "trying to wrap [his] head around the mro > computation in 2.2". Apparently there is the algorithm mentioned at > http://www.python.org/2.2.1/descrintro.html (dubbed the naive algorithm) > and then the one implemented in typeobject.c_ (called the 2.2 algorithm). > Samuele discovered some inconsistencies with the implemented algorithm > that he desired some explanation about. > > Guido responded, thankful that someone was giving this a look because his > "intuition about the equivalence between algorithms turned out to be > wrong". Guido stated that he thought that he wrote the algorithm from the > book "Putting Metaclasses To Work" correctly sans raising an error when > major conflicts occur in the ordering. In a later email Guido explained > that the naive algorithm came about by his attempt to simplify the > explanation of the 2.2 algorithm. Guido pretty much wrote the algorithm > from the aforementioned book. Now the algorithm is not simple, so Guido > did his best to simplify the explanation. Unknowningly, though, he came > up with a variant on the algorithm in his explanation. > > Greg Ewing pointed out that he thought the naive algorithm was nicer since > it seemed to work more intuitively and was easier to explain (and remember > kids, these are basic tenants in Python programming). Guido ended up > stating that "If Samuele agrees that the naive algorithm works better, > [Guido will] try to make it so in 2.3". Well, Samuele said that the "2.2 > mro is the worst of our options". > > There was a problem, though, with the naive algorithm; it is not monotonic > as pointed out by Samuele. This led him to put out two options: > > 1. Use the naive algorithm, which had the drawback of being monotonic. s/being/not being/ > Samuele also believed that it didn't produce "the most natural results". > > 2. Adopt C3_ as described at > http://www.webcom.com/haahr/dylan/linearization-oopsla96.html and > apparently used by Dylan_ and Goo_ . only by Goo. Dylan uses stilll a different algorithm with results more similar to those of the one used by CLOS, but I don't think that's important. They are all described in the paper. Dylan would have adopted C3, but it was switching algorithm aftering starting with the CLOS one and having already code around. > This algorithm is monotonic and > Samuele says is more intuitive. more intuitive in its results. > Guido got around to reading the C3_ paper and agreed that "we should adopt > C3". > He thought that the 2.2 algorithm was like the C*[LOOPS] algorithm > mentioned in the paper. there was a typo, it's L*[LOOPS]. Btw, we still don't know exactly, it's probably a variant, and I don't think it's so relevant. > Samuele then wrote a C implementation of the > algorithm. Guido said he would get to the patch after 2.2.2b1 got out the > door. > > .. _C3: http://www.webcom.com/haahr/dylan/linearization-oopsla96.html > .. _Dylan: http://monday.sourceforge.net/wiki/ if the reference to Dylan goes away above, this one should also go away. Otherwise I would use http://www.gwydiondylan.org/ as reference URL for Dylan or simply comp.lang.dylan . > .. _Goo: http://www.ai.mit.edu/~jrb/goo/manual.43/goomanual_55.html > > .._typeobject.c: > http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Objects/t ypeobject.c regards. From martin@v.loewis.de Sun Oct 13 20:48:08 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: Sun, 13 Oct 2002 21:48:08 +0200 Subject: [Python-Dev] The -U flag Message-ID: <200210131948.g9DJm8E2005271@mira.informatik.hu-berlin.de> Since Python allows Unicode strings in sys.path, we are making progress on getting -U to work. When entering interactive mode, the stumbling block is _idmap = '' for i in range(256): _idmap = _idmap + chr(i) del i Here, _idmap is initialized with a Unicode string, and the chr(i) results are promoted to Unicode, which eventually causes a UnicodeErorr when you get past 127. The work-around would be to write _idmap = str('') for i in range(256): _idmap = _idmap + chr(i) del i With that, we can enter interactive mode in python -U. Is such a change acceptable? Curious, Martin P.S. In case you wonder how the testsuite works for -U: The first problem is File "/usr/src/python/Lib/warnings.py", line 267, in ? filterwarnings("ignore", category=OverflowWarning, append=1) File "/usr/src/python/Lib/warnings.py", line 131, in filterwarnings assert isinstance(message, str), "message must be a string" AssertionError: message must be a string I'm not going to ask whether I can use basestring there, yet... From tim.one@comcast.net Sun Oct 13 21:07:39 2002 From: tim.one@comcast.net (Tim Peters) Date: Sun, 13 Oct 2002 16:07:39 -0400 Subject: [Python-Dev] Why does the Windows installer use the shortname formof the install directory? In-Reply-To: <20021013112136.GA77207@hishome.net> Message-ID: [Christian Tismer] > The F1 key gives you a regular windows help file, as I remember. I hate "regular Windows help files", but I agree it exists . > And you can surround the > Convert to short filename > thing with an if contruct that checks for a space, something like > If MAINDIR Contains " " then > # here the Convert thingie > End [Oren Tirosh] > Translated to WISEspeak: > > Parse String "%MAINDIR%" into BEFORESPACE and AFTERSPACE [*] > If AFTERSPACE Not Equal "" then > Set Variable MAINDIR to %MAINDIR% > End Christian's was already WISEspeak, but, as before, I've got insufficent interest to bother with any of this. If one of you wants to submit a tested patch, great! From amk@nyman.amk.ca Sun Oct 13 22:38:38 2002 From: amk@nyman.amk.ca (A.M. Kuchling) Date: Sun, 13 Oct 2002 17:38:38 -0400 Subject: [Python-Dev] PEP 218 update questions Message-ID: <200210132138.g9DLccX06379@nyman.amk.ca> I've just updated PEP 218 to match the sets.py module that's in CVS. The original PEP text had a single Set class, and Set instances froze after their __hash__ was called so you could use a set as a dictionary key. sets.py takes a more straightforward approach, with distinct Set and ImmutableSet classes. Two things in the PEP need to be updated to take this change into account. 1) The long-term proposal in PEP 218 lists a single built-in conversion function, set(iterable). This made sense when there was a single Set class, but now this needs to be updated. How would instances of a built-in immutable set type be created? Would there be a second immutable_set() built-in, or would the set() function take an additional argument: set(iterable, immutable=True)? 2) The PEP proposes {1,2,3} as the set notation and {-} for the empty set. Would there be different syntax for an immutable and a mutable set? My thought is, "no"; {1,2,3} or {-} is always mutable, and an immutable set would be written as immutable_set({1,2,3}). That's up to python-dev, though, but whatever gets chosen needs to be specified in the PEP. --amk From drifty@bigfoot.com Sun Oct 13 22:43:27 2002 From: drifty@bigfoot.com (Brett Cannon) Date: Sun, 13 Oct 2002 14:43:27 -0700 (PDT) Subject: [Python-Dev] PEP 218 update questions In-Reply-To: <200210132138.g9DLccX06379@nyman.amk.ca> Message-ID: [A.M. Kuchling] > How would instances of a built-in immutable set type be created? > Would there be a second immutable_set() built-in, or would the set() > function take an additional argument: set(iterable, immutable=True)? > I say have a single set() with an argument flag. Mutable or not, they are both subsets of BaseSet and thus should be reflected in the constructor function. > 2) > The PEP proposes {1,2,3} as the set notation and {-} for the empty > set. Would there be different syntax for an immutable and a mutable > set? > As long as mutable sets can be automatically converted to immutable, I say keep only one syntax. If the user wants to explicitly create an immutable set, call set(mutable_set, immutable=True) or have a very explicit way for mutable sets to turn themselves into immutable sets. No need to add more syntax for something that can automatically be done for the user. -Brett C. From aleaxit@yahoo.com Sun Oct 13 23:05:44 2002 From: aleaxit@yahoo.com (Alex Martelli) Date: Mon, 14 Oct 2002 00:05:44 +0200 Subject: [Python-Dev] PEP 218 update questions In-Reply-To: References: Message-ID: <02101400054404.01541@arthur> On Sunday 13 October 2002 23:43, Brett Cannon wrote: > [A.M. Kuchling] > > > How would instances of a built-in immutable set type be created? > > Would there be a second immutable_set() built-in, or would the set() > > function take an additional argument: set(iterable, immutable=True)? > > I say have a single set() with an argument flag. Mutable or not, they > are both subsets of BaseSet and thus should be reflected in the > constructor function. Why expose a function rather than exposing the type directly? I think we should have Set, the type, rather than set, the function, as the built-in -- so it can be inherited from as well as being constructed. This, in turn, suggests exposing type ImmutableSet too. That is, if we want such built-ins at all. If (e.g.) types such as array and mmap can stay in modules, why not sets, for now? > > 2) > > The PEP proposes {1,2,3} as the set notation and {-} for the empty > > set. Would there be different syntax for an immutable and a mutable > > set? > > As long as mutable sets can be automatically converted to immutable, I > say keep only one syntax. If the user wants to explicitly create an > immutable set, call set(mutable_set, immutable=True) or have a very > explicit way for mutable sets to turn themselves into immutable sets. No > need to add more syntax for something that can automatically be done for > the user. Either via ImmutableSet, or the method currently private and named _as_immutable could be made public and named as_immutable or asImmutable. I prefer the latter solution because it would give us a needed protocol -- "return an immutable snapshot of your current state" -- sets now do it, but it would be nice to have a way for all mutable types that are interested in cooperating in such "snapshot" operations to be able to (lists could return tuple(self), dicts could return 'frozendicts' if and when such beasts appear, etc) in such a uniform way. One last possibility, there could instead be a builtin immutable(x) (or immutable_copy(x)) that knows about some builtin types (e.g. turns lists into tuples itself, returns already-immutable types unchanged) and tries to delegate to a new special method __as_immutable__ -- this would be at least as good as the second solution above and more in keeping with how several existing Python builtins already work. Alex From arigo@ulb.ac.be Sun Oct 13 23:23:34 2002 From: arigo@ulb.ac.be (Armin Rigo) Date: Mon, 14 Oct 2002 00:23:34 +0200 Subject: [Python-Dev] Memory size overflows References: Message-ID: <3DA9F266.C87E5A58@ulb.ac.be> Hello Tim, Let me quote from stringobject.c the "heartbreaking code such as string_repeat's": nbytes = size * sizeof(char); if (nbytes / sizeof(char) != (size_t)size || nbytes + sizeof(PyStringObject) <= nbytes) op = (PyStringObject *) PyObject_MALLOC(sizeof(PyStringObject) + nbytes); if (op == NULL) With the proper macro, the whole code above can be written as: nbytes = Py_MUL_ADD(size, sizeof(char), sizeof(PyStringObject)); op = (PyStringObject *) PyObject_MALLOC(nbytes); if (op == NULL) which seems much cleaner to me. For this reason I do not believe it is a good idea here to define a macro a la Zope C: DO_MULADD_OR(nbytes, size, sizeof(char), sizeof(PyStringObject), goto Overflow); for two reasons: there is no real need for a special "Overflow" case, as long as the returned 'nbytes' is guaranteed to be un-malloc-able; besides, no existing macro (e.g. _PyObject_VAR_SIZE) can support an extra "goto Overflow" case without breaking C code that depends on it. There is however a difference between the two above code fragments, in that the former will raise an OverflowError for "really too large values" and a MemoryError for "reasonable value but still too large for me to allocate right now". Whether this is a correct behavior is debatable. I would argue that raising MemoryError in both cases is fine: after all, "123"*9999999999999999 could be perfectly fine in another Python implementation with lazy strings. It is thus only a memory limit that makes this fail in the current implementation. Note that if we define MAX_SIZE to ((size_t)-1), the overflow check in the present stringobject.c could be more efficiently written down as: if (size > (MAX_SIZE - sizeof(PyStringObject)) / sizeof(char)) where everything at the right of '>' reduces to a compile-time constant. The Py_MUL_ADD macro can be implemented with this idea in mind, which works best if the 2nd and 3rd arguments are constants. > note that anything/non_constant can be incredibly expensive! Yes. Couldn't a better overflow-checking algorithm be designed for these cases? For example, to multiply the non-constants 'a' and 'b', we can be sure that there is no overflow if both 'a' and 'b' are small enough (which is the common case anyway). I believe I can even think about a single Py_MUL_ADD macro that is as efficient as above with constant arguments, and otherwise uses this trick. About using 64-bit computations everywhere (as suggested by Christian): if the length of objects (e.g. lists) is still constrained to 31 bits, then it's fine; otherwise, the problem is the same. Do all supported 64-bit platforms have a 32-bit 'int', and thus a 31-bit limit on object lengths? By the way, this makes me think of a good efficient overflow-detection method for platforms with a "long long" type longer than size_t: unsigned long long bytes = ((unsigned long long)a)*((unsigned long long)b)+c; if (bytes > ((size_t)-1)) ; This compiles to very good assembly code on x86. Mmmh, in summary I would recommend that we define a common "multiply-and-add" macro, returning a size_t or OVERFLOW_VALUE in case of overflow, which is implemented in several different ways on different platforms, as selected by #if's. The same macro is used for just "multiply" or just "add" by relying on the compiler's strengh reduction. Armin From drifty@bigfoot.com Mon Oct 14 00:11:12 2002 From: drifty@bigfoot.com (Brett Cannon) Date: Sun, 13 Oct 2002 16:11:12 -0700 (PDT) Subject: [Python-Dev] PEP 218 update questions In-Reply-To: <02101400054404.01541@arthur> Message-ID: [Alex Martelli] > Why expose a function rather than exposing the type directly? I think > we should have Set, the type, rather than set, the function, as the > built-in -- so it can be inherited from as well as being constructed. That is what I meant; function is a factory function like dict() and friends are now. Sorry if that wasn't clear. > This, in turn, suggests exposing type ImmutableSet too. > I don't see that as an automatic conclusion. I prefer to view ImmutableSet as just as a Set with a property turned on. But tuple and lists do set a precedent for clearly separating a mutable and immutable version of something. Probably should stick with that separation. > > As long as mutable sets can be automatically converted to immutable, I > > say keep only one syntax. If the user wants to explicitly create an > > immutable set, call set(mutable_set, immutable=True) or have a very > > explicit way for mutable sets to turn themselves into immutable sets. No > > need to add more syntax for something that can automatically be done for > > the user. > > Either via ImmutableSet, or the method currently private and named > _as_immutable could be made public and named as_immutable or > asImmutable. I prefer the latter solution because it would give us a > needed protocol -- "return an immutable snapshot of your current > state" -- sets now do it, but it would be nice to have a way for all > mutable types that are interested in cooperating in such "snapshot" > operations to be able to (lists could return tuple(self), dicts could > return 'frozendicts' if and when such beasts appear, etc) in such a > uniform way. One last possibility, there could instead be a builtin > immutable(x) (or immutable_copy(x)) that knows about some builtin > types (e.g. turns lists into tuples itself, returns already-immutable > types unchanged) and tries to delegate to a new special method > __as_immutable__ -- this would be at least as good as the second > solution above and more in keeping with how several existing Python > builtins already work. > I really like that idea. Being able to use practically anything as a dict key would be rather nice. There doesn't need to be a guarantee that the copy will keep up with the original, mutable object, just that it corresponds at the immutable object's creation. That eliminates that possible hornet's nest. If we could come up with an immutable dict (I suspect just removing __setitem__() doesn't cut it =) we could take that and be able to create immutable object copies by just making a snapshot of the object's __dict__, __bases__, and such. But the big use I see of an immutable_copy() function is being able to take parameters and making sure you don't change them in-place. You can, of course, just make sure you don't change them, but this could eliminate having to watch out for that and let you worry about other things. -Brett From guido@python.org Mon Oct 14 00:48:33 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 13 Oct 2002 19:48:33 -0400 Subject: [Python-Dev] The -U flag In-Reply-To: Your message of "Sun, 13 Oct 2002 21:48:08 +0200." <200210131948.g9DJm8E2005271@mira.informatik.hu-berlin.de> References: <200210131948.g9DJm8E2005271@mira.informatik.hu-berlin.de> Message-ID: <200210132348.g9DNmXH09873@pcp02138704pcs.reston01.va.comcast.net> > Since Python allows Unicode strings in sys.path, we are making > progress on getting -U to work. When entering interactive mode, the > stumbling block is > > _idmap = '' > for i in range(256): _idmap = _idmap + chr(i) > del i > > Here, _idmap is initialized with a Unicode string, and the chr(i) > results are promoted to Unicode, which eventually causes a > UnicodeErorr when you get past 127. > > The work-around would be to write > > _idmap = str('') > for i in range(256): _idmap = _idmap + chr(i) > del i > > With that, we can enter interactive mode in python -U. > > Is such a change acceptable? Only with a comment that explains it -- otherwise the next person looking at the code will remove it. But didn't we at one point conclude that -U was never gonna work? And wasn't that why it's no longer documented? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 14 00:52:05 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 13 Oct 2002 19:52:05 -0400 Subject: [Python-Dev] PEP 218 update questions In-Reply-To: Your message of "Sun, 13 Oct 2002 17:38:38 EDT." <200210132138.g9DLccX06379@nyman.amk.ca> References: <200210132138.g9DLccX06379@nyman.amk.ca> Message-ID: <200210132352.g9DNq5N09900@pcp02138704pcs.reston01.va.comcast.net> > I've just updated PEP 218 to match the sets.py module that's in CVS. Thanks! > The original PEP text had a single Set class, and Set instances froze > after their __hash__ was called so you could use a set as a dictionary > key. sets.py takes a more straightforward approach, with distinct Set > and ImmutableSet classes. > > Two things in the PEP need to be updated to take this change into account. > > 1) > The long-term proposal in PEP 218 lists a single built-in conversion > function, set(iterable). This made sense when there was a single Set > class, but now this needs to be updated. > > How would instances of a built-in immutable set type be created? > Would there be a second immutable_set() built-in, or would the set() > function take an additional argument: set(iterable, immutable=True)? > > 2) > The PEP proposes {1,2,3} as the set notation and {-} for the empty > set. Would there be different syntax for an immutable and a mutable > set? > > My thought is, "no"; {1,2,3} or {-} is always mutable, and an > immutable set would be written as immutable_set({1,2,3}). That's up > to python-dev, though, but whatever gets chosen needs to be specified > in the PEP. I don't want to think about this right now -- and it's not yet relevant since I expect this won't become an built-in type for a couple more releases. Can't you just add these questions to the PEP as unresolved issues? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 14 00:55:32 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 13 Oct 2002 19:55:32 -0400 Subject: [Python-Dev] PEP 218 update questions In-Reply-To: Your message of "Sun, 13 Oct 2002 16:11:12 PDT." References: Message-ID: <200210132355.g9DNtWr09924@pcp02138704pcs.reston01.va.comcast.net> > I don't see that as an automatic conclusion. I prefer to view > ImmutableSet as just as a Set with a property turned on. But it supports a different set of methods! I prefer to view it as a separate type. --Guido van Rossum (home page: http://www.python.org/~guido/) From Raymond Hettinger" Message-ID: <002501c27316$06f21b80$680ca044@oemcomputer> > 1) > The long-term proposal in PEP 218 lists a single built-in conversion > function, set(iterable). This made sense when there was a single Set > class, but now this needs to be updated. > > How would instances of a built-in immutable set type be created? > Would there be a second immutable_set() built-in, or would the set() > function take an additional argument: set(iterable, immutable=True)? > -1 on the property version. As it stands now, immutables can be constructed directly from a class with its own methods or, more likely, is an automagically created proxy used internally by mutuable sets. I think of immutable sets as regular sets that are frozen for purposes of inclusion in another set -- there is almost no reason for instantiatiating them directly (other than to save coercion time for sets whose contents are known and fixed at the time of instatiation). > 2) > The PEP proposes {1,2,3} as the set notation and {-} for the empty > set. Would there be different syntax for an immutable and a mutable > set? > > My thought is, "no"; {1,2,3} or {-} is always mutable, and an > immutable set would be written as immutable_set({1,2,3}). That's up > to python-dev, though, but whatever gets chosen needs to be specified > in the PEP. +1. Immutable sets have their own weirdness. No use reflecting that in the core language syntax. Besides, they may yet evolve further (perhaps with an animmutable.refresh() method). Raymond Hettinger From tdelaney@avaya.com Mon Oct 14 01:27:18 2002 From: tdelaney@avaya.com (Delaney, Timothy) Date: Mon, 14 Oct 2002 10:27:18 +1000 Subject: [Python-Dev] Python-dev Summary for 2002-10-01 through 2002-1 0-13 Message-ID: > From: Brett Cannon [mailto:bac@OCF.Berkeley.EDU] > > .. _frameobject.c: > http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/d > ist/src/Objects/frameobject.c Might be worthwhile always including a TinyURL as well as the full URL to deal with wrapping issues (I would suggested a Shorter Link, but Alex Martelli showed me a better one). For example, the above URL gets changed to: http://tinyurl.com/1ynh so the resulting entry would look like .. _frameobject.c: http://tinyurl.com/1ynh http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Object s/frameobject.c It also suggests that it may be worthwhile special-casing the reST parser for this case (and http://makeashorterlink.com/ ... any others?). Tim Delaney From drifty@bigfoot.com Mon Oct 14 01:57:36 2002 From: drifty@bigfoot.com (Brett Cannon) Date: Sun, 13 Oct 2002 17:57:36 -0700 (PDT) Subject: [Python-Dev] Python-dev Summary for 2002-10-01 through 2002-1 0-13 In-Reply-To: Message-ID: [Delaney, Timothy] > > From: Brett Cannon [mailto:bac@OCF.Berkeley.EDU] > > Might be worthwhile always including a TinyURL as well as the full URL to > deal with wrapping issues (I would suggested a Shorter Link, but Alex > Martelli showed me a better one). > > For example, the above URL gets changed to: > > http://tinyurl.com/1ynh > > so the resulting entry would look like > > .. _frameobject.c: http://tinyurl.com/1ynh > http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Object > s/frameobject.c > I could do that, but I have two concerns. One is the extra amount of time it woulkd take to do links for every long link (tend to be a decent amount), and there is no guarantee that tinyurl will be around and thus make the archived links break. Now I know that the long links will break at some point too, but I would like to minimize that problem. And I realize putting both in solves this, but I just don't love the idea that half of all the links (redundent or not) could go bad because of one site (if python.org goes down, Python will have gone down so I am not going to worry about that). -Brett From martin@v.loewis.de Mon Oct 14 05:32:14 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 14 Oct 2002 06:32:14 +0200 Subject: [Python-Dev] The -U flag In-Reply-To: <200210132348.g9DNmXH09873@pcp02138704pcs.reston01.va.comcast.net> References: <200210131948.g9DJm8E2005271@mira.informatik.hu-berlin.de> <200210132348.g9DNmXH09873@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > But didn't we at one point conclude that -U was never gonna work? I'm not aware of a formal decision about that. I readily admit that it wont' work in any foreseeable future, but I haven't given up for P3k :-) This time, my interest was triggered by someone on c.l.p who could not reverse a string, since it was in UTF-8; he was asking why Python is so difficult for Swedish people. > And wasn't that why it's no longer documented? Yes: because it won't work in any foreseeable future. I'll take your statement that "small" changes towards making -U work are acceptable; I'll see how far I get with small changes until I get stuck again. Regards, Martin From eggsy@gmx.de Mon Oct 14 05:50:07 2002 From: eggsy@gmx.de (Eric Johnson) Date: Mon, 14 Oct 2002 06:50:07 +0200 (MEST) Subject: [Python-Dev] Missing symbol in sre.py (Python 2.2.2b1) Message-ID: <20478.1034571007@www43.gmx.net> There appears to still be a missing public symbol definition in the Python 2.2.2b1 version of sre.py The documentation implies that finditer should work and indeed it is defined in the module if sys.hexversion > 0x02020000 however it is not listed in the __all__ module variable. This means that if I use import re I am unable to use re.finditer but if I use import sre then I can use sre.finditer. I thought that the recommended approach was to use import re. I think this is the outstanding bug 585882 re.finditer which was for Python 2.2.1. Eric -- +++ GMX - Mail, Messaging & more http://www.gmx.net +++ NEU: Mit GMX ins Internet. Rund um die Uhr für 1 ct/ Min. surfen! From aleaxit@yahoo.com Mon Oct 14 07:17:58 2002 From: aleaxit@yahoo.com (Alex Martelli) Date: Mon, 14 Oct 2002 08:17:58 +0200 Subject: [Python-Dev] PEP 218 update questions In-Reply-To: References: Message-ID: <02101408175804.02129@arthur> On Monday 14 October 2002 01:11, Brett Cannon wrote: > [Alex Martelli] > > > Why expose a function rather than exposing the type directly? I think > > we should have Set, the type, rather than set, the function, as the > > built-in -- so it can be inherited from as well as being constructed. > > That is what I meant; function is a factory function like dict() and > friends are now. Sorry if that wasn't clear. I think of functions (including factory ones) and types (or classes) as separate, e.g. because I can subclass the latter but not the former. But the docs sometimes aren't crystal clear on that because what used to be factory functions (e.g. int) have recently become types. > > This, in turn, suggests exposing type ImmutableSet too. > > I don't see that as an automatic conclusion. I prefer to view > ImmutableSet as just as a Set with a property turned on. I think in Python that wouldn't work, because both Set and ImmutableSet have methods the other shouldn't have -- so if you inherit you'd better know which one you want, since removing methods during inheritance is near-unfeasible. > But tuple and lists do set a precedent for clearly separating a mutable > and immutable version of something. Probably should stick with that > separation. However tuples aren't "immutable lists" -- they don't have non-mutating methods that lists do have, for example. So the precedent's iffy. > I really like that idea. Being able to use practically anything as a > dict key would be rather nice. There doesn't need to be a guarantee that > the copy will keep up with the original, mutable object, just that it > corresponds at the immutable object's creation. That eliminates that > possible hornet's nest. The needed axiom is that the original and copy are == at creation time, and loosely speaking that apart from mutability they are more or less equivalent objects in some sense or other (e.g. a mutable one may be rebuilt from the immutable one later) > If we could come up with an immutable dict (I suspect just removing > __setitem__() doesn't cut it =) we could take that and be able > to create immutable object copies by just making a snapshot of the > object's __dict__, __bases__, and such. > > But the big use I see of an immutable_copy() function is being able to > take parameters and making sure you don't change them in-place. You can, > of course, just make sure you don't change them, but this could eliminate > having to watch out for that and let you worry about other things. For that you can use copy.copy or if need be copy.deepcopy -- i.e. you may not care if the function makes changes, as long as it's on a copy, not on the original. BTW this suggests that copy.immutablecopy, not a builtin, is the right place to put the new hypothetical functionality (still probably cooperating with an optional special method, like copy.copy and copy.deepcopy do). Would be weird if the function to make ordinary copies lived one place and the one to make immutable copies lived somewhere else, I think. BTW, I removed: "A.M. Kuchling" from the Cc since it's bouncing... Alex From mal@lemburg.com Mon Oct 14 08:37:06 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 14 Oct 2002 09:37:06 +0200 Subject: [Python-Dev] The -U flag References: <200210131948.g9DJm8E2005271@mira.informatik.hu-berlin.de> <200210132348.g9DNmXH09873@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3DAA7422.8000703@lemburg.com> Guido van Rossum wrote: >>Since Python allows Unicode strings in sys.path, we are making >>progress on getting -U to work. When entering interactive mode, the >>stumbling block is >> >>_idmap = '' >>for i in range(256): _idmap = _idmap + chr(i) >>del i >> >>Here, _idmap is initialized with a Unicode string, and the chr(i) >>results are promoted to Unicode, which eventually causes a >>UnicodeErorr when you get past 127. >> >>The work-around would be to write >> >>_idmap = str('') >>for i in range(256): _idmap = _idmap + chr(i) >>del i >> >>With that, we can enter interactive mode in python -U. >> >>Is such a change acceptable? > > > Only with a comment that explains it -- otherwise the next person > looking at the code will remove it. Wouldn't the following be faster ? l = map(chr, range(256)) _idmap = str('').join(l) del l > But didn't we at one point conclude that -U was never gonna work? And > wasn't that why it's no longer documented? I don't remember such a conclusion. The reason for having -U in the first place was to test the Python standard lib for being able to handle Unicode seemlessly. Obviously we are still far from having reached that point, but it's a good way of being able to test the compatibility. It's no longer documented to reduce the number of bug-reports we get for it. It was never meant to be used by users (at this point), just by developers. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From sjoerd@acm.org Mon Oct 14 08:57:05 2002 From: sjoerd@acm.org (Sjoerd Mullender) Date: Mon, 14 Oct 2002 09:57:05 +0200 Subject: [Python-Dev] Why does the Windows installer use the shortname formof the install directory? In-Reply-To: References: Message-ID: <200210140757.g9E7v5H06545@indus.ins.cwi.nl> On Fri, Oct 11 2002 Tim Peters wrote: > [Moore, Paul] > > ... > > That's fair enough. And it's not major enough that I'm pushing > > for anyone to waste time on it. I just thought I'd mention it in > > case it was a simple change to a checkbox, or something. > > It *almost* is a checkbox! In WISE's idea of "a scripting language", the > relevant line looks like this: > > Set Variable MAINDIR to %MAINDIR% > > There's not a visible clue that this isn't a no-op. If you double-click on > the line in the WISE GUI, though, a dialog box pops up, with a dropdown list > labeled "Operation", one of whose entries is > > Convert to short filename > > There isn't another variant that adds "but only if it contains a space". > > Beyond that, I'm stuck -- since Adobe graciously updated my Acrobat Reader, > every attempt to view the WISE reference docs (pdf format) hard-freezes > (power cycle required) my box. I *like* your suggestion to skip this unless > there's an embedded space, I just seem unable to do anything about it now > (but it would still be for 2.3 -- mucking with 2.2 is too dangerous). You should be able to put an If Statement / End Statement around it with the If Statement property box containing If Variable: MAINDIR Contains The Value: . I haven't actually tested this, but I have used this statement with other values. -- Sjoerd Mullender From guido@python.org Mon Oct 14 13:16:30 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 14 Oct 2002 08:16:30 -0400 Subject: [Python-Dev] The -U flag In-Reply-To: Your message of "Mon, 14 Oct 2002 06:32:14 +0200." References: <200210131948.g9DJm8E2005271@mira.informatik.hu-berlin.de> <200210132348.g9DNmXH09873@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210141216.g9ECGU611229@pcp02138704pcs.reston01.va.comcast.net> > > But didn't we at one point conclude that -U was never gonna work? > > I'm not aware of a formal decision about that. I readily admit that it > wont' work in any foreseeable future, but I haven't given up for P3k :-) Maybe MAL remembers after seeing this checkin comment to Modules/main.c: ---------------------------- revision 1.63 date: 2002/02/11 18:46:47; author: lemburg; state: Exp; lines: +0 -1 Remove mentioning of -U option in "python -h" output. ---------------------------- > This time, my interest was triggered by someone on c.l.p who could not > reverse a string, since it was in UTF-8; he was asking why Python is > so difficult for Swedish people. Yeah, he mailed me first -- I don't understand why his default encoding isn'y Latin-1 though. > > And wasn't that why it's no longer documented? > > Yes: because it won't work in any foreseeable future. > > I'll take your statement that "small" changes towards making -U work > are acceptable; I'll see how far I get with small changes until I get > stuck again. Sure. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 14 13:23:06 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 14 Oct 2002 08:23:06 -0400 Subject: [Python-Dev] Missing symbol in sre.py (Python 2.2.2b1) In-Reply-To: Your message of "Mon, 14 Oct 2002 06:50:07 +0200." <20478.1034571007@www43.gmx.net> References: <20478.1034571007@www43.gmx.net> Message-ID: <200210141223.g9ECN6W11276@pcp02138704pcs.reston01.va.comcast.net> > There appears to still be a missing public symbol definition in the Python > 2.2.2b1 > version of sre.py > > The documentation implies that finditer should work and indeed it is > defined > in the module if sys.hexversion > 0x02020000 however it is not listed in > the > __all__ module variable. > > This means that if I use > import re > I am unable to use re.finditer but if I use > import sre > then I can use sre.finditer. > > I thought that the recommended approach was to use import re. > > I think this is the outstanding bug 585882 re.finditer which was for > Python 2.2.1. Thanks! I fixed this just in time for the release. I don't know what Fredrik is doing these days, but fixing bugs in RE doesn't seem to be a high priority for him... :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 14 13:39:10 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 14 Oct 2002 08:39:10 -0400 Subject: [Python-Dev] Python 2.2.2 release tonight - checkin freeze on branch Message-ID: <200210141239.g9ECdAC11799@pcp02138704pcs.reston01.va.comcast.net> I'd like to do the 2.2.2 release tonight, unless Jack has any unresolved problems. (Jack, mail me ASAP if you do!) Please, no more checkins to the release22-maint branch until the release is done (with the exception of the Mac subtree, and Fred & Tim). I still have to update the Misc/NEWS file. --Guido van Rossum (home page: http://www.python.org/~guido/) From Jack.Jansen@cwi.nl Mon Oct 14 14:07:34 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Mon, 14 Oct 2002 15:07:34 +0200 Subject: [Python-Dev] Python 2.2.2 release tonight - checkin freeze on branch In-Reply-To: <200210141239.g9ECdAC11799@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Monday, Oct 14, 2002, at 14:39 Europe/Amsterdam, Guido van Rossum wrote: > I'd like to do the 2.2.2 release tonight, unless Jack has any > unresolved problems. (Jack, mail me ASAP if you do!) There's fix for test_email (skip it for MacPython) that's still sitting on my disk at home, but I'd say don't let that stop you. And it may be a while before I can actually release MacPython 2.2.2, because I'm still waiting for a new freeware license for the VISE installer. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From pinard@iro.umontreal.ca Mon Oct 14 14:45:48 2002 From: pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois?= Pinard) Date: Mon, 14 Oct 2002 09:45:48 -0400 Subject: [Python-Dev] Re: Assign to errno allowed? In-Reply-To: <200209260059.g8Q0xmL06185@oma.cosc.canterbury.ac.nz> (Greg Ewing's message of "Thu, 26 Sep 2002 12:59:48 +1200 (NZST)") References: <200209260059.g8Q0xmL06185@oma.cosc.canterbury.ac.nz> Message-ID: [Greg Ewing] >> #ifdef ...WINCE... > ^^^^^ > > I wonder if Microsoft foresaw that abbreviation when > they chose the name CE... Speaking of, for many GNU projects, we needed a pre-processor symbol to represent DJGPP and other ports to MS-DOS and Windows, so someone (most probably me) took `DOS' and `WIN' and concatenated them, yielding: #if DOSWIN ... It went for a few years. One day, we were asked by the FSF to avoid this symbol, as it suggested that DOS may win in some way. At the time, this surely meant a lot of places to change, I do not remember if we complied. The fact is that I never had such a pun in head when I made the symbol... -- François Pinard http://www.iro.umontreal.ca/~pinard From nas@python.ca Mon Oct 14 15:05:03 2002 From: nas@python.ca (Neil Schemenauer) Date: Mon, 14 Oct 2002 07:05:03 -0700 Subject: [Python-Dev] Python-dev Summary for 2002-10-01 through 2002-1 0-13 In-Reply-To: References: Message-ID: <20021014140503.GA27628@glacier.arctrix.com> Brett Cannon wrote: > I could do that, but I have two concerns. One is the extra amount of time > it woulkd take to do links for every long link (tend to be a decent > amount), and there is no guarantee that tinyurl will be around and thus > make the archived links break. There's no reason we couldn't implement the same service on python.org. Neil From guido@python.org Mon Oct 14 15:28:49 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 14 Oct 2002 10:28:49 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Your message of "Mon, 14 Oct 2002 00:23:34 +0200." <3DA9F266.C87E5A58@ulb.ac.be> References: <3DA9F266.C87E5A58@ulb.ac.be> Message-ID: <200210141428.g9EESnv12174@pcp02138704pcs.reston01.va.comcast.net> I'd like to add that, until we decide on a Really Big Rewrite, the length of all Python sequences must fit in a signed int. So relying on malloc returning NULL isn't really a viable approach: even if you have, say, 2.5 GB of memory in your box, you still can't create a string of that size. I do agree that the distinction between OverflowError and MemoryError is not worth splitting hairs over. --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Mon Oct 14 18:34:15 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 14 Oct 2002 19:34:15 +0200 Subject: [Python-Dev] Missing symbol in sre.py (Python 2.2.2b1) References: <20478.1034571007@www43.gmx.net> <200210141223.g9ECN6W11276@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <01fd01c273a8$93561f30$ced241d5@hagrid> guido wrote: > I don't know what Fredrik is doing these days, but fixing bugs in RE > doesn't seem to be a high priority for him... :-( sorry, life got in the way. and I have to admit that hacking on RE hasn't resulted in enough sex, praise or money [1] lately to keep me motivated. hopefully, that'll change in time for 2.3... cheers /F 1) http://blogs.salon.com/0001111/2002/09/21.html#a475 From martin@v.loewis.de Mon Oct 14 19:05:57 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 14 Oct 2002 20:05:57 +0200 Subject: [Python-Dev] The -U flag In-Reply-To: <200210141216.g9ECGU611229@pcp02138704pcs.reston01.va.comcast.net> References: <200210131948.g9DJm8E2005271@mira.informatik.hu-berlin.de> <200210132348.g9DNmXH09873@pcp02138704pcs.reston01.va.comcast.net> <200210141216.g9ECGU611229@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > Yeah, he mailed me first -- I don't understand why his default > encoding isn'y Latin-1 though. He uses an sv_SE.UTF-8 locale, which appears to be the default on Redhat 8, if you ask for a Swedish environment. IMO, UTF-8 locales will replace everything else on Linux in the long run. Otherwise, system-wide files (and file names) will be restricted to ASCII forever, and people will not accept not having accented and CJK characters in /etc/passwd, for their real names. I expect other Unices to take a similar direction. Regards, Martin From guido@python.org Mon Oct 14 19:10:35 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 14 Oct 2002 14:10:35 -0400 Subject: [Python-Dev] The -U flag In-Reply-To: Your message of "Mon, 14 Oct 2002 20:05:57 +0200." References: <200210131948.g9DJm8E2005271@mira.informatik.hu-berlin.de> <200210132348.g9DNmXH09873@pcp02138704pcs.reston01.va.comcast.net> <200210141216.g9ECGU611229@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210141810.g9EIAZ124143@pcp02138704pcs.reston01.va.comcast.net> > IMO, UTF-8 locales will replace everything else on Linux in the long > run. Otherwise, system-wide files (and file names) will be restricted > to ASCII forever, and people will not accept not having accented and > CJK characters in /etc/passwd, for their real names. I expect other > Unices to take a similar direction. We'll be in for lots of niggling little bug reports then, as RH 8.0 conquers the world... :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Mon Oct 14 19:20:05 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 14 Oct 2002 14:20:05 -0400 Subject: [Python-Dev] The -U flag In-Reply-To: <200210141216.g9ECGU611229@pcp02138704pcs.reston01.va.comcast.net> Message-ID: > Maybe MAL remembers after seeing this checkin comment to Modules/main.c: > > ---------------------------- > revision 1.63 > date: 2002/02/11 18:46:47; author: lemburg; state: Exp; lines: +0 -1 > Remove mentioning of -U option in "python -h" output. > ---------------------------- MAL did that after I whined a lot: so long as -U was mentioned in the -h blurb, users kept trying to use it -- that it didn't work, and spectacularly so, gave them a wrong impression about Python, and wasted time on various mailing lists explaining over & over that it wasn't "supposed to" work. From martin@v.loewis.de Mon Oct 14 21:04:26 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 14 Oct 2002 22:04:26 +0200 Subject: [Python-Dev] The -U flag In-Reply-To: <3DAA7422.8000703@lemburg.com> References: <200210131948.g9DJm8E2005271@mira.informatik.hu-berlin.de> <200210132348.g9DNmXH09873@pcp02138704pcs.reston01.va.comcast.net> <3DAA7422.8000703@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > Wouldn't the following be faster ? > > l = map(chr, range(256)) > _idmap = str('').join(l) > del l It is, by a factor of roughly 3 (on my system). Committed as posted. Regards, Martin From BPettersen@NAREX.com Mon Oct 14 23:30:58 2002 From: BPettersen@NAREX.com (Bjorn Pettersen) Date: Mon, 14 Oct 2002 16:30:58 -0600 Subject: [Python-Dev] Python 2.2.2 release tonight - checkin freeze on branch Message-ID: <60FB8BB7F0EFC7409B75EEEC13E2019201773D50@admin56.narex.com> > On Monday, Oct 14, 2002, at 14:39 Europe/Amsterdam, Guido van Rossum=20 > wrote: >=20 > > I'd like to do the 2.2.2 release tonight, unless Jack has any=20 > > unresolved problems. (Jack, mail me ASAP if you do!) I realize this is a little late, but I was wondering if 2.2.2 will support VC7 (important for us Python-in-a-tie people ;-) I searched the mailing list, but the only thing I found were messages that it was fixed on the 2.3 branch... -- bjorn From guido@python.org Mon Oct 14 23:38:34 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 14 Oct 2002 18:38:34 -0400 Subject: [Python-Dev] Python 2.2.2 release tonight - checkin freeze on branch In-Reply-To: Your message of "Mon, 14 Oct 2002 16:30:58 MDT." <60FB8BB7F0EFC7409B75EEEC13E2019201773D50@admin56.narex.com> References: <60FB8BB7F0EFC7409B75EEEC13E2019201773D50@admin56.narex.com> Message-ID: <200210142238.g9EMcZB01255@pcp02138704pcs.reston01.va.comcast.net> > I realize this is a little late, but I was wondering if 2.2.2 will > support VC7 (important for us Python-in-a-tie people ;-) I searched > the mailing list, but the only thing I found were messages that it > was fixed on the 2.3 branch... By VC7, do you refer to Microsoft Visual C++ 7.0? I don't have a copy, and neither does Tim. Maybe the PBF can donate one? --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Mon Oct 14 23:44:31 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 14 Oct 2002 17:44:31 -0500 Subject: [Python-Dev] Can Zope be built on MacOSX? Message-ID: <15787.18639.177235.144826@montanaro.dyndns.org> I downloaded Zope 2.6.0b1 today from zope.org and tried to build it with Python from CVS, which failed, then Python 2.2.2 (also from CVS, but which should be between 2.2.2b1 and 2.2.2final), which also failed. Upon closer examination, it seems 2.1.x is required for Zope 2.5, though the doc/INSTALL.txt file says nothing about what Zope 2.6 requires. I then pulled the release21-maint branch of CVS and tried to build it on MacOSX. It failed spectacularly trying to compile Python/dynload_next.c Am I wasting my time trying to build Zope on MacOSX? Skip From lalo@laranja.org Mon Oct 14 23:47:46 2002 From: lalo@laranja.org (Lalo Martins) Date: Mon, 14 Oct 2002 20:47:46 -0200 Subject: [Python-Dev] Can Zope be built on MacOSX? In-Reply-To: <15787.18639.177235.144826@montanaro.dyndns.org> References: <15787.18639.177235.144826@montanaro.dyndns.org> Message-ID: <20021014224746.GQ10221@laranja.org> On Mon, Oct 14, 2002 at 05:44:31PM -0500, Skip Montanaro wrote: > > I downloaded Zope 2.6.0b1 today from zope.org and tried to build it with > Python from CVS, which failed, then Python 2.2.2 (also from CVS, but which > should be between 2.2.2b1 and 2.2.2final), which also failed. Upon closer > examination, it seems 2.1.x is required for Zope 2.5, though the > doc/INSTALL.txt file says nothing about what Zope 2.6 requires. I then > pulled the release21-maint branch of CVS and tried to build it on MacOSX. > It failed spectacularly trying to compile Python/dynload_next.c > > Am I wasting my time trying to build Zope on MacOSX? This is not the correct list for this question, but then again I think you know that quite well ;-) Zope requires 2.1, this includes Zope 2.6. The first Python2.2 version is scheduled to be 2.7. I don't have OSX, but I heard about some difficulties. My coworker however successfully compiled 2.1.3 (from the source release, not CVS) and Z2.6 from CVS and reports it to work without a problem. He recommends latest gcc3. []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ Python Foundry Guide http://www.sf.net/foundry/python-foundry/ From guido@python.org Mon Oct 14 23:56:12 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 14 Oct 2002 18:56:12 -0400 Subject: [Python-Dev] Can Zope be built on MacOSX? In-Reply-To: Your message of "Mon, 14 Oct 2002 17:44:31 CDT." <15787.18639.177235.144826@montanaro.dyndns.org> References: <15787.18639.177235.144826@montanaro.dyndns.org> Message-ID: <200210142256.g9EMuC208037@pcp02138704pcs.reston01.va.comcast.net> [Skip] > I downloaded Zope 2.6.0b1 today from zope.org and tried to build it > with Python from CVS, which failed, then Python 2.2.2 (also from > CVS, but which should be between 2.2.2b1 and 2.2.2final), which also > failed. Upon closer examination, it seems 2.1.x is required for > Zope 2.5, though the doc/INSTALL.txt file says nothing about what > Zope 2.6 requires. I then pulled the release21-maint branch of CVS > and tried to build it on MacOSX. It failed spectacularly trying to > compile Python/dynload_next.c I'm hardly an expert on building Zope on MacOSX, but it's my understanding that Zope 2.6 ought to work with Python 2.2.2 (as well as 2.2.2b1) even though this is not officially supported. Ditto for CVS trunk ("Python 2.3"). I can also believe that Python 2.1.3 doesn't build on MacOSX -- MacOSX simply didn't exist when we released 2.1. AFAIK Jack hasn't really backported all the work he did for MacOS to 2.1 -- and I doubt he ever will. > Am I wasting my time trying to build Zope on MacOSX? I wouldn't say that. What failures did you see with ~2.2.2? (Maybe take this offline or to a Zope list?) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Oct 15 00:29:33 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 14 Oct 2002 19:29:33 -0400 Subject: [Python-Dev] RELEASED: Python 2.2.2 (final) Message-ID: <200210142329.g9ENTXH13982@pcp02138704pcs.reston01.va.comcast.net> Ta-daa! The release of Python 2.2.2 is here. This is a fully backwards compatible bugfix release for Python 2.2.1. Thanks to all who submitted last-minute bug reports and fixes -- and thanks even more to those who didn't wait until the last minute! :-) Python 2.2.2 has a large number of bug fixes in all areas of the interpreter and the library; for a full account see the release notes, available via the link below. With special dispensation from the "bugfixes only" rule, a brand new version of the email package is also included: email 2.4.3. The new release is available here: http://www.python.org/2.2.2/ It will soon be available from mirror sites all over the world -- for a speedy download, try the mirror site nearest to you first: http://www.python.org/Mirrors.html As usual, a Windows installer and a Unix/Linux source tarball are made available, as well as tarballs of the documentation in various forms. At the moment, no Mac version is available, though I expect that Jack Jansen will produce one soon. Likewise, no Linux RPMs are available yet; I expect that Sean Reifschneider will prepare these shortly. I'm especially grateful to Michael Hudson, Raymond Hettinger and Neal Norwitz for their hard work on backporting the large volume of patches from Python 2.3. Without them this release would not have been possible. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Tue Oct 15 02:40:34 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 14 Oct 2002 21:40:34 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: <3DA9F266.C87E5A58@ulb.ac.be> Message-ID: [Armin Rigo] > Let me quote from stringobject.c the "heartbreaking code such as > string_repeat's": > > nbytes = size * sizeof(char); > if (nbytes / sizeof(char) != (size_t)size || > nbytes + sizeof(PyStringObject) <= nbytes) > > op = (PyStringObject *) > PyObject_MALLOC(sizeof(PyStringObject) + nbytes); > if (op == NULL) > > > With the proper macro, the whole code above can be written as: > > nbytes = Py_MUL_ADD(size, sizeof(char), sizeof(PyStringObject)); > op = (PyStringObject *) PyObject_MALLOC(nbytes); > if (op == NULL) > > > which seems much cleaner to me. It's not the appearance of the code that breaks my heart (although it doesn't cheer it either ), it's that when we "fix" one of these things that pops up in real life, we add a pile of expensive tests and branches to code that's been known to fall over only once in 10 years. So Python gets bigger and slower for everyone, and half the time there's no real cause beyond that some wiener is playing "let's try to break it!". > For this reason I do not believe it is a good idea here to define a > macro a la Zope C: > > DO_MULADD_OR(nbytes, size, sizeof(char), sizeof(PyStringObject), > goto Overflow); > > for two reasons: there is no real need for a special "Overflow" case, I don't know. It depends on context, such as how many times a potential overflow can occur within a single routine. Probably not often on average. > as long as the returned 'nbytes' is guaranteed to be un-malloc-able; There is no nbytes that's guaranteed to bu un-malloc-able, unless you're going to impose artificial restrictions on malloc. It's not a happy thing to burden our malloc wrappers with another layer of test-and-branch either. > besides, no existing macro (e.g. _PyObject_VAR_SIZE) can support an > extra "goto Overflow" case without breaking C code that depends on it. Cool: yet another reason to leave things alone . > There is however a difference between the two above code fragments, in > that the former will raise an OverflowError for "really too large > values" and a MemoryError for "reasonable value but still too large for > me to allocate right now". Whether this is a correct behavior is > debatable. Some things aren't worth debating, though -- it doesn't matter to me which exception we raise. > I would argue that raising MemoryError in both cases is fine: Fine by me too. > after all, "123"*9999999999999999 could be perfectly fine in another > Python implementation with lazy strings. It is thus only a > memory limit that makes this fail in the current implementation. > > Note that if we define MAX_SIZE to ((size_t)-1), the overflow check in > the present stringobject.c could be more efficiently written down as: (size_t)~0UL is safer, although there's really no wholly safe way to spell the intent in C89. > if (size > (MAX_SIZE - sizeof(PyStringObject)) / sizeof(char)) > > > where everything at the right of '>' reduces to a compile-time > constant. The Py_MUL_ADD macro can be implemented with this idea in > mind, which works best if the 2nd and 3rd arguments are constants. >> note that anything/non_constant can be incredibly expensive! > Yes. Couldn't a better overflow-checking algorithm be designed for > these cases? For example, to multiply the non-constants 'a' and 'b', we > can be sure that there is no overflow if both 'a' and 'b' are small > enough (which is the common case anyway). I believe I can even think > about a single Py_MUL_ADD macro that is as efficient as above with > constant arguments, and otherwise uses this trick. And more tests, and more branches. Life will be much easier if we can limit it to size_t thingies (which have the same size, and are unsigned0. > About using 64-bit computations everywhere (as suggested by Christian): > if the length of objects (e.g. lists) is still constrained to 31 bits, > then it's fine; otherwise, the problem is the same. Do all supported > 64-bit platforms have a 32-bit 'int', and thus a 31-bit limit on object > lengths? No. For example, some now-rare models of Cray boxes have no 32-bit types. If 64-bit machines really are the wave of the future (I started using them in the late 1970's, and am still waiting for the revolution <0.9 wink>), it's no cure anyway, as Python in a truly 64-bit world will have to lose its 2GB-bounded of the world (as it's already painfully doing for files, and maybe for mmaps). Then the "detect overflow" problem returns with a vengeance. > By the way, this makes me think of a good efficient overflow-detection > method for platforms with a "long long" type longer than size_t: > > unsigned long long bytes = ((unsigned long long)a)*((unsigned long > long)b)+c; > if (bytes > ((size_t)-1)) > ; This also implicitly assumes that: 1. size_t is no wider than an int; that's the kind of thing the "(size_t)~0UL would be safer" comment above was getting at. 2. long long is at least twice as wide as size_t. > This compiles to very good assembly code on x86. > > Mmmh, in summary I would recommend that we define a common > "multiply-and-add" macro, returning a size_t or OVERFLOW_VALUE in case > of overflow, which is implemented in several different ways on different > platforms, as selected by #if's. The same macro is used for just > "multiply" or just "add" by relying on the compiler's strengh reduction. What a PITA. Is it worth it? If I don't have to do it, maybe . BTW, when we allocate an object that participates in cyclic GC, there's Yet Another little integer that gets added to the total size (to reflect the "extra header" bytes unique to GC'ed objects), and so there's Yet Another place potential overflow may go uncaught now. From tim.one@comcast.net Tue Oct 15 02:43:55 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 14 Oct 2002 21:43:55 -0400 Subject: [Python-Dev] Missing symbol in sre.py (Python 2.2.2b1) In-Reply-To: <01fd01c273a8$93561f30$ced241d5@hagrid> Message-ID: [Fredrik Lundh] > sorry, life got in the way. > > and I have to admit that hacking on RE hasn't resulted in enough > sex, praise or money [1] lately to keep me motivated. hopefully, > that'll change in time for 2.3... > > cheers /F > > 1) http://blogs.salon.com/0001111/2002/09/21.html#a475 Here you go, /F: real, actual praise -- just like the fellow asked for. What more could you want? Swedes give all their money to the government, and working on RE is plain incompatible with having orgasms . From guido@python.org Tue Oct 15 05:24:56 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 15 Oct 2002 00:24:56 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Your message of "Mon, 14 Oct 2002 21:40:34 EDT." References: Message-ID: <200210150424.g9F4Ous17546@pcp02138704pcs.reston01.va.comcast.net> I think this discussion is suffering from the desire to over-generalize. Armin, can you show us how to multiply two nonnegative ints together with an overflow "block" as Tim suggested earlier? I.e. a macro callable as SAFE_MULTIPLY(destination, src1, src2, on_overflow); meaning roughly destination = src1 * src2; if () on_overflow; I currently use something like this as the overflow test in a few places: if (src1 && destination / src1 != src2) but as Tim points out the division can kill us unless src1 is a constant... How would you spell this without doing a division? --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Tue Oct 15 06:29:36 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 15 Oct 2002 01:29:36 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: <200210150424.g9F4Ous17546@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido] > ... > I currently use something like this as the overflow test in a few places: > > if (src1 && destination / src1 != src2) > > but as Tim points out the division can kill us unless src1 is a > constant... How would you spell this without doing a division? Armin answered that in the case there's an efficient "unsigned long long" type at least twice the width of the inputs. Then you just do the product in the fat precision and see whether any of the upper-half bits are set. As a pragmatic matter, 2/5ths of that trick can work "most of the time": #define YOWZA (sizeof(size_t) * 4U) if ((src1 | src2} >> YOWZA) hard to tell else overflow is impossible would get out quick in *most* non-overflow cases. The fast path there succeeds if and only if both inputs are at most "half full". For example, in string * int it's rare that len(string) >= 64K or int >= 64K. For what to do in the other half, see the history of subtly buggy attempts to detect overflow quickly in Python's int * int code. Without a double-width type to fall back on, I think it's impossible to do this both correctly and quickly. BTW, if we know we're not interested in anything other than 32-bit products, double _prod = (double)src1 * (double)src2; if (_prod < 4294967296.0) /* 2**32 */ result = (size_t)_prod; else overflow; does the trick reliably on all platforms I know of -- although not all Python platforms support floating-point efficiently! It's a good puzzle . From thomas.heller@ion-tof.com Tue Oct 15 11:02:56 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: 15 Oct 2002 12:02:56 +0200 Subject: [Python-Dev] tempfile problems Message-ID: <3cr8j9yn.fsf@ion-tof.com> bdist_wininst currently contains this code: from tempfile import NamedTemporaryFile arc = NamedTemporaryFile(".zip") archive_basename = arc.name[:-4] fullname = self.distribution.get_fullname() arcname = self.make_archive(archive_basename, "zip", root_dir=self.bdist_dir) The corresponding checkin message is this: revision 1.35 date: 2002/08/09 16:37:34; author: gvanrossum; state: Exp; lines: +5 -4 Massive changes from SF 589982 (tempfile.py rewrite, by Zack Weinberg). This changes all uses of deprecated tempfile functions to the recommended ones. The call to NamedTemporaryFile(".zip") crashes with an access violation on Windows. I've submitted a bug # 623464 for this. I assume this is meant instead: from tempfile import NamedTemporaryFile arc = NamedTemporaryFile(suffix = ".zip") archive_basename = arc.name[:-4] fullname = self.distribution.get_fullname() arcname = self.make_archive(archive_basename, "zip", root_dir=self.bdist_dir) but it doesn't work either: make_archive() fails with 'permission denied' because the file is already open. This would work: from tempfile import NamedTemporaryFile arc = NamedTemporaryFile(suffix = ".zip") archive_basename = arc.name[:-4] arc.close() fullname = self.distribution.get_fullname() arcname = self.make_archive(archive_basename, "zip", root_dir=self.bdist_dir) but I'm not sure if it defeats the purpose of the change (secure tempfiles). Since distutils should still be compatible with older Python versions, and those do not contain NamedTemporaryFile, I suggest to back out this change. Thomas From mal@lemburg.com Tue Oct 15 11:09:55 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 15 Oct 2002 12:09:55 +0200 Subject: [Python-Dev] tempfile problems References: <3cr8j9yn.fsf@ion-tof.com> Message-ID: <3DABE973.2080601@lemburg.com> Thomas Heller wrote: > bdist_wininst currently contains this code: > > from tempfile import NamedTemporaryFile > arc = NamedTemporaryFile(".zip") > archive_basename = arc.name[:-4] > fullname = self.distribution.get_fullname() > arcname = self.make_archive(archive_basename, "zip", > root_dir=self.bdist_dir) > > The corresponding checkin message is this: > > revision 1.35 > date: 2002/08/09 16:37:34; author: gvanrossum; state: Exp; lines: +5 -4 > Massive changes from SF 589982 (tempfile.py rewrite, by Zack > Weinberg). This changes all uses of deprecated tempfile functions to > the recommended ones. > > > The call to NamedTemporaryFile(".zip") crashes with an access > violation on Windows. I've submitted a bug # 623464 for this. > I assume this is meant instead: > > from tempfile import NamedTemporaryFile > arc = NamedTemporaryFile(suffix = ".zip") > archive_basename = arc.name[:-4] > fullname = self.distribution.get_fullname() > arcname = self.make_archive(archive_basename, "zip", > root_dir=self.bdist_dir) > > but it doesn't work either: make_archive() fails with > 'permission denied' because the file is already open. > This would work: > > from tempfile import NamedTemporaryFile > arc = NamedTemporaryFile(suffix = ".zip") > archive_basename = arc.name[:-4] > arc.close() > fullname = self.distribution.get_fullname() > arcname = self.make_archive(archive_basename, "zip", > root_dir=self.bdist_dir) > > but I'm not sure if it defeats the purpose of the change > (secure tempfiles). > > Since distutils should still be compatible with older > Python versions, and those do not contain NamedTemporaryFile, > I suggest to back out this change. +1 A temporary ZIP file doesn't introduce much of a security risk anyway, even less when it's being created by a package developer. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From Paolo Invernizzi Tue Oct 15 11:58:04 2002 From: Paolo Invernizzi (Paolo Invernizzi) Date: Tue, 15 Oct 2002 12:58:04 +0200 Subject: Re[2]: [Python-Dev] Python 2.2.2 release tonight - checkin freeze on branch In-Reply-To: <60FB8BB7F0EFC7409B75EEEC13E2019201773D50@admin56.narex.com> References: <60FB8BB7F0EFC7409B75EEEC13E2019201773D50@admin56.narex.com> Message-ID: <13914848100.20021015125804@dmsware.com> BP> I realize this is a little late, but I was wondering if 2.2.2 will BP> support VC7 (important for us Python-in-a-tie people ;-) I've just tried... Python 2.2.2 does not compile with Visual Studio NET out-of-the-box... It complains about missing largeint.h, so the fixes Mark (was Mark?) has done on 2.3 weren't backported to 2.2.2. I haven't looked at it, but think that also the DLL_EXPORT/IMPORT stuff is still broken on 2.2.2 -- Best regards, Paolo From xscottg@yahoo.com Tue Oct 15 18:24:09 2002 From: xscottg@yahoo.com (Scott Gilbert) Date: Tue, 15 Oct 2002 10:24:09 -0700 (PDT) Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: <20021015172409.50190.qmail@web40110.mail.yahoo.com> The following is really ugly (until you put it in a macro), but would work: #define HALF_BITS (sizeof(size_t) * 4U) #define LO_MASK ((((size_t)1) << HALF_BITS)-1) #define HI_MASK (LO_MASK<>HALF_BITS)) & HI_MASK) | (((src1>>HALF_BITS) * (src2&LO_MASK)) & HI_MASK) | (src1>>HALF_BITS) * (src2>>HALF_BITS)) ) { goto overflow; } Of course it's hoping for some common subexpression elimination from the compiler. __________________________________________________ Do you Yahoo!? Faith Hill - Exclusive Performances, Videos & More http://faith.yahoo.com From jason@tishler.net Tue Oct 15 18:51:32 2002 From: jason@tishler.net (Jason Tishler) Date: Tue, 15 Oct 2002 13:51:32 -0400 Subject: [Python-Dev] Dropping support for Tcl 8.0 and 8.1 In-Reply-To: <200210011519.g91FJwI21585@odiug.zope.com> References: <200210010607.g9167Mns001471@mira.informatik.hu-berlin.de> <200210011338.g91Dclo08817@odiug.zope.com> <200210011519.g91FJwI21585@odiug.zope.com> Message-ID: <20021015175131.GB1536@tishler.net> Martin, On Tue, Oct 01, 2002 at 11:19:58AM -0400, Guido van Rossum wrote: > > > If it's more work to keep support around than to drop it, sure. > > > > I'm planning a number of changes which can be simplified if I don't > > have that many versions to support. > > OK, +1. This is probably too little, too late, but Cygwin is still stuck at Tcl 8.0. Would you be willing to keep this support in? Note that I've tried to encourage Cygwin to upgrade to a more recent version: http://cygwin.com/ml/cygwin/2002-10/msg00798.html Thanks, Jason From jason@tishler.net Tue Oct 15 19:02:33 2002 From: jason@tishler.net (Jason Tishler) Date: Tue, 15 Oct 2002 14:02:33 -0400 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: <200210050018.g950IFO32089@pcp02138704pcs.reston01.va.comcast.net> References: <200210041940.g94Jeph29549@pcp02138704pcs.reston01.va.comcast.net> <20021004235039.GA97477@gargamel.ghaering.test> <200210050018.g950IFO32089@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021015180232.GC1536@tishler.net> On Fri, Oct 04, 2002 at 08:18:15PM -0400, Guido van Rossum wrote: > We've already absorbed a sheer endless set of patches to make it work > on CYGWIN. Ouch! :,) Nevertheless, the Cygwin community appreciates the continued support. Jason From tim.one@comcast.net Tue Oct 15 21:36:19 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 15 Oct 2002 16:36:19 -0400 Subject: [Python-Dev] [win32] Killing MSVC's _alloca In-Reply-To: <20021015180232.GC1536@tishler.net> Message-ID: [Guido] > We've already absorbed a sheer endless set of patches to make it work > on CYGWIN. [Jason Tishler] > Ouch! :,) > > Nevertheless, the Cygwin community appreciates the continued support. The Python community appreciates your support too, Jason! Guido wasn't being pejorative there, just objectively accurate: "sheer endless" is Dutch for "about 12" . From martin@v.loewis.de Tue Oct 15 21:57:58 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 15 Oct 2002 22:57:58 +0200 Subject: [Python-Dev] Dropping support for Tcl 8.0 and 8.1 In-Reply-To: <20021015175131.GB1536@tishler.net> References: <200210010607.g9167Mns001471@mira.informatik.hu-berlin.de> <200210011338.g91Dclo08817@odiug.zope.com> <200210011519.g91FJwI21585@odiug.zope.com> <20021015175131.GB1536@tishler.net> Message-ID: Jason Tishler writes: > This is probably too little, too late, but Cygwin is still stuck at Tcl > 8.0. Would you be willing to keep this support in? No, I've already made the changes to drop this support, and added support for UCS-4 mode in _tkinter. > Note that I've tried to encourage Cygwin to upgrade to a more recent > version: > > http://cygwin.com/ml/cygwin/2002-10/msg00798.html That seems the right way to go. They may have changed Cygwin by the time Python 2.3 is released. If not, users wishing to use Cygwin _tkinter can use Mumit Khan's Tcl 8.3/8.4 ports: ftp://ftp.nanotech.wisc.edu/pub/khan/tcl/tcltk-8.3.4-cygwin/ Regards, Martin From tim.one@comcast.net Tue Oct 15 23:25:27 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 15 Oct 2002 18:25:27 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: <20021015172409.50190.qmail@web40110.mail.yahoo.com> Message-ID: [Scott Gilbert] > The following is really ugly (until you put it in a macro), but > would work: > > #define HALF_BITS (sizeof(size_t) * 4U) > #define LO_MASK ((((size_t)1) << HALF_BITS)-1) > #define HI_MASK (LO_MASK< > if ( > /* Tim's fast path */ > ((src1 | src2) & HI_MASK) && ( > /* Ugly slower path */ > (((src1&LO_MASK) * (src2>>HALF_BITS)) & HI_MASK) | > (((src1>>HALF_BITS) * (src2&LO_MASK)) & HI_MASK) | > (src1>>HALF_BITS) * (src2>>HALF_BITS)) > ) { > goto overflow; > } I'll repeat that it's hard to do this correctly and quickly (c.f. earlier comments about the history of Python's subtly buggy attempts). This kind of approach is viewing src1 and src2 as 2-digit numbers in base X = 2**HALF_BITS: src1 = A*X + B src2 = C*X + D then viewing their product as highpart: A*C*X**2 + xxxxxxxx middlepart1: A*D*X + xxxxxxxx middlepart2: B*C*X + xxxxxxxx lowpart: B*D xxxxxxxx ---------------- xxxxxxxxxxxxxxxx < safe > Your second line checks whether the high half of middlepart2 is 0; your third line whether the high half of middlepart1 is 0; your last line whether highpart is 0. But they can all be 0 and *still* have overflow: the sum low half of middlepart1 + low half of middlepart2 + high half of lowpart can carry over into the zone. Supposing we were looking at 16-bit products, here's an example: 0x01ff * 0x00ff ------ 0000 00ff 0000 fe01 -------- 0001fd01 ^ oops Hint: before posting something like this, prototype it in Python, and do an exhaustive check of all possible pairs of multiplicands in an artificially small base. Very few buggy algorithms survive that, even for very small artificial bases. From guido@python.org Wed Oct 16 01:47:17 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 15 Oct 2002 20:47:17 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? Message-ID: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> I just found a place in my own code that broke because stat now returns floats for mtime etc. I was using a Zope "OIBTree" object, which is a dict-like extension type mapping strings to ints; it very reasonbly doesn't like it if you assign a float to it. I'm not say that because *my* code breaks, this is a bad feature. But I am worried that we'll find more programs using external data types that won't accept floats (I'm thinking of Numpy arrays), and I'm now not so sure if this is acceptable breakage... Apparently the fact that it "works" on the Mac isn't really a great proof. Maybe we can introduce a variant of the stat() function that returns floats, or alternative field names that are only available when using attributes (not when using the tuple-ish API)? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 16 02:46:32 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 15 Oct 2002 21:46:32 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Your message of "Tue, 15 Oct 2002 01:29:36 EDT." References: Message-ID: <200210160146.g9G1kWn20397@pcp02138704pcs.reston01.va.comcast.net> > if we know we're not interested in anything other than 32-bit products, > > double _prod = (double)src1 * (double)src2; > if (_prod < 4294967296.0) /* 2**32 */ > result = (size_t)_prod; > else > overflow; > > does the trick reliably on all platforms I know of -- although not > all Python platforms support floating-point efficiently! I like this one, as well as the one using 64-bit precision. I wonder if we could collect a bunch of different macros and make them tunable for different platforms? (Including a "don't check for overflow" version for people who wanna play fast and loose. :-) The trick will to ensure that the macro arguments are used exactly once in all versions (otherwise switching macros might unveal a new class of bugs). Apart from *how* to check, there's a variety of cases depending on the types of the input and output variables. There's the ob_size calculation for sequence repeat, which takes a C int and a C long and should return something that fits in a C int; and there's the malloc size calculation for various ones, which should take an int and a (small constant) size_t and return a size_t. So maybe we need two macros: one for int*long->int, and one for int*size_t->size_t? With several different implementations, chosen by configure. But first we'd have to see of there are enough places to benefit from such elaborate machinery. There are also additions, usually of small constants, which generally operate on the size_t. These are easier to test: if you add a small positive constant to a size_t, and the result is smaller than the original size_t, an overflow occurred. Similar for plain ints. So I'm not sure we need much help with these. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Wed Oct 16 03:09:47 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 15 Oct 2002 22:09:47 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15788.51819.500992.678200@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> Maybe we can introduce a variant of the stat() function that GvR> returns floats, or alternative field names that are only GvR> available when using attributes (not when using the tuple-ish GvR> API)? +1 on the latter idea. Since the st_mtime, st_atime, and st_ctime attributes, and the tuple-ish API are documented as returning ints, I think you shouldn't change that. Provide different field names for float values, e.g. f_mtime, f_ctime, f_atime, or maybe just mtime, atime, and ctime... ? -Barry From xscottg@yahoo.com Wed Oct 16 04:41:20 2002 From: xscottg@yahoo.com (Scott Gilbert) Date: Tue, 15 Oct 2002 20:41:20 -0700 (PDT) Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: <20021016034120.46693.qmail@web40101.mail.yahoo.com> [Tim Peters] > > I'll repeat that it's hard to do this correctly and quickly (c.f. earlier > comments about the history of Python's subtly buggy attempts). > [...] > > 0x01ff * > 0x00ff > ------ > 0000 > 00ff > 0000 > fe01 > -------- > 0001fd01 > ^ oops > That was sloppy on my part. Thanks for showing me my error. (That's as gracious as I can be while saying "D'Oh!" on the inside. :-) Clearly you know the long answer to this problem. It could involve taking the ugly expression I had and adding 3 more terms to see if it adds up to a problem. Or if all those integer multiplies make it less attractive than just going to double, then taking the log2() of each operand and counting the bits to see if there is a problem. Or using the PyLong_* machinery. Or something similar. And you've got your fast path past the test for the common case, so it looks like you're just hesitant to make the big ugly macro... __________________________________________________ Do you Yahoo!? Faith Hill - Exclusive Performances, Videos & More http://faith.yahoo.com From greg@cosc.canterbury.ac.nz Wed Oct 16 06:13:04 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Oct 2002 18:13:04 +1300 (NZDT) Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: <200210160513.g9G5D4003141@oma.cosc.canterbury.ac.nz> Tim Peters : > I'll repeat that it's hard to do this correctly and quickly (c.f. earlier > comments about the history of Python's subtly buggy attempts). The really tragic thing about all this is that most machines have perfectly good facilities for detecting overflow at the machine level, but no commonly used HLL that I know of provides access to them, for some unfathomable reason. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one@comcast.net Wed Oct 16 06:29:18 2002 From: tim.one@comcast.net (Tim Peters) Date: Wed, 16 Oct 2002 01:29:18 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: <200210160513.g9G5D4003141@oma.cosc.canterbury.ac.nz> Message-ID: [Greg Ewing] > The really tragic thing about all this is that most > machines have perfectly good facilities for detecting > overflow at the machine level, but no commonly used > HLL that I know of provides access to them, for some > unfathomable reason. Well, that's where it bifurcates Yet Again -- Pentiums have a perfectly usable "give me the high bits of a 32x32->64 bit product" instruction, and it's just a couple lines of inline assembler under MSVC or gcc (both make such stuff easy to get at, embedded in C code). But as I said at the start, I ignore these things until they pop up in real life. So few *have* popped in real life that there hasn't been any point to creating mounds of platform-dependent #ifdef'ed infrastructure. From tismer@tismer.com Wed Oct 16 06:36:41 2002 From: tismer@tismer.com (Christian Tismer) Date: Tue, 15 Oct 2002 22:36:41 -0700 Subject: [Python-Dev] Stackless for 2.2.2 ready! Message-ID: <3DACFAE9.60905@tismer.com> Hi Python-community, The Stackless Python beta has been ported to the Python 2.2.2 release, one day after the announce. It could have been even faster, but I had to work a lot on typeobject.c, which had been backported from 2.3. My own patches to that are now very tiny and will be submitted, soon. Changes to Stackless: The option to build statically has been re-enabled. Supported platforms are now Windows, Mac OS X, Linux-X86, Cygwin, SunOS Sparc with gcc, PPC Unix, S390 & S390x. A scheduler object with overridable strategies has been prepared (functionality will be added these days) Metatypes can have slots (se above). See http://www.stackless.com/ for details, CVS access etc. Binaries for Python 2.2.2 have been built and include _tkinter. CVS access to the last 2.2.1 version is possible, check out with the tag "last_2_2_1". good luck - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tim.one@comcast.net Wed Oct 16 06:44:02 2002 From: tim.one@comcast.net (Tim Peters) Date: Wed, 16 Oct 2002 01:44:02 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: <200210160146.g9G1kWn20397@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido] > ... > There are also additions, usually of small constants, which generally > operate on the size_t. These are easier to test: if you add a small > positive constant to a size_t, and the result is smaller than the > original size_t, an overflow occurred. More generally, if x and y are both size_t, and sizeof(size_t) >= sizeof(unsigned int), x+y overflowed if and only if x+y < x Checking x+y < y is equivalent -- you can compare the sum to either input; doesn't matter. > Similar for plain ints. Signed ints are harder. If the signs differ, overflow is impossible. If the signs are the same, overflow occurred if and only if the sign of the result differs from the common input sign. sum = x+y overflow iff ((sum ^ x) & (sum ^ y)) < 0 > So I'm not sure we need much help with these. Well, they're all obscure as hell. From martin@v.loewis.de Wed Oct 16 06:57:17 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 16 Oct 2002 07:57:17 +0200 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: <15788.51819.500992.678200@gargle.gargle.HOWL> References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <15788.51819.500992.678200@gargle.gargle.HOWL> Message-ID: barry@python.org (Barry A. Warsaw) writes: > +1 on the latter idea. Since the st_mtime, st_atime, and st_ctime > attributes, and the tuple-ish API are documented as returning ints, I > think you shouldn't change that. Provide different field names for > float values, e.g. f_mtime, f_ctime, f_atime, or maybe just mtime, > atime, and ctime... ? The previous suggestion was st_mtimef, st_ctimef, ... Regards, Martin From martin@v.loewis.de Wed Oct 16 07:08:29 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 16 Oct 2002 08:08:29 +0200 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > I just found a place in my own code that broke because stat now > returns floats for mtime etc. I was using a Zope "OIBTree" object, > which is a dict-like extension type mapping strings to ints; it > very reasonbly doesn't like it if you assign a float to it. In this case, could you fix your code by truncating/rounding, or by using a different container? > Maybe we can introduce a variant of the stat() function that returns > floats, or alternative field names that are only available when using > attributes (not when using the tuple-ish API)? Assuming that in all cases of breakage it would be possible to easily "fix" the code somehow, I'd like to have a migration plan, following PEP 5: I.e. together with adding a new feature for float time stamps, we should deprecate the old feature (of int time stamps) - perhaps not right now, but as a pending deprecation. Then, after some time, we should drop the old feature and continue with just the new one. So I'd like to see a solution which does not look ugly when the deprecated feature is gone and just the new feature stays. Perhaps a flag "times_are_float" would be the right approach? N: flag defaults to False, users need to ask for floats explicitly N+1: omitting the flag is deprecated N+2: flag defaults to True, users need to ask for ints explicitly N+5: the flag is removed There are variations to this plan, adding a phase where it is an error to omit the flag, or where a pending deprecation is added. Regards, Martin From mal@lemburg.com Wed Oct 16 08:09:13 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 16 Oct 2002 09:09:13 +0200 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3DAD1099.1040507@lemburg.com> Guido van Rossum wrote: > I just found a place in my own code that broke because stat now > returns floats for mtime etc. I was using a Zope "OIBTree" object, > which is a dict-like extension type mapping strings to ints; it > very reasonbly doesn't like it if you assign a float to it. > > I'm not say that because *my* code breaks, this is a bad feature. But > I am worried that we'll find more programs using external data types > that won't accept floats (I'm thinking of Numpy arrays), and I'm now > not so sure if this is acceptable breakage... Apparently the fact > that it "works" on the Mac isn't really a great proof. > > Maybe we can introduce a variant of the stat() function that returns > floats, or alternative field names that are only available when using > attributes (not when using the tuple-ish API)? If you're suggesting a new API, why not have it return one of the new datetime objects ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From arigo@tunes.org Wed Oct 16 09:11:18 2002 From: arigo@tunes.org (Armin Rigo) Date: Wed, 16 Oct 2002 10:11:18 +0200 (CEST) Subject: [Python-Dev] Memory size overflows In-Reply-To: <200210160146.g9G1kWn20397@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Hello Guido, On Tue, 15 Oct 2002, Guido van Rossum wrote: > I like this one, as well as the one using 64-bit precision. I wonder > if we could collect a bunch of different macros and make them tunable > for different platforms? Doable. I experimented with some macros that do (unsigned int)*(unsigned int)->size_t, where the macro to use is selected according to #if's based on the relative size of int, long and long long. > (...) The trick will > to ensure that the macro arguments are used exactly once in all > versions (otherwise switching macros might unveal a new class of > bugs). Harder... > Apart from *how* to check, there's a variety of cases depending on the > types of the input and output variables. True as well... Why are C macros so limited?... > There are also additions, usually of small constants, which generally > operate on the size_t. If we are sure these are small, we can just add without any check if we reserve a whole range of values for "overflow", e.g. ((size_t)~0x7fff) to ((size_t)~0). Not too clean however. Armin From oren-py-d@hishome.net Wed Oct 16 09:15:32 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Wed, 16 Oct 2002 04:15:32 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: References: <200210150424.g9F4Ous17546@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021016081532.GA49294@hishome.net> On Tue, Oct 15, 2002 at 01:29:36AM -0400, Tim Peters wrote: > BTW, if we know we're not interested in anything other than 32-bit products, > > double _prod = (double)src1 * (double)src2; > if (_prod < 4294967296.0) /* 2**32 */ > result = (size_t)_prod; > else > overflow; s/_prod;/src1 * src2;/ Re-calculating the product with integer multiplication is probably faster than converting a double to integer on any sane CPU. I wonder if the old SPARCs that don't have an integer multiply instruction actually have a floating point to integer conversion that is faster than integer mul. Wasting "expensive" multiplications this way still feels blasphemous :) Oren From recomnd@pandanic.com Wed Oct 16 08:25:35 2002 From: recomnd@pandanic.com (pandaNIC) Date: Wed, 16 Oct 2002 15:25:35 +0800 Subject: [Python-Dev] Recommend "Register domain with only $14.95" Message-ID: <2002101672535654CBOLDNQEHKHE@M5Mailer> This is a multi-part message in MIME format --=_NextPart_2rfkindysadvnqw3nerasdf Content-Type: text/plain;charset="windows-1252" Content-Transfer-Encoding: quoted-printable Somebody send you this mail to recommend our domain registration service= : register/renew or transfer domain (com/net/org/biz/info/us) with only $1= 4.95 per year. go to http://www.pandanic.com for more details, and check out the other = services, such as email/url forwarding and DNS manager. (this mail will be sent only by once) ---------------------------------------- http://www.pandanic.com --=_NextPart_2rfkindysadvnqw3nerasdf Content-Type: text/html; charset="windows-1252" Content-Transfer-Encoding: quoted-printable
 
Somebody send you this mail to recommend our domain registration=20= service:
 
 
register/renew or transfer domain (com/net/org/biz/info/us) with on= ly=20 $14.95 per year.
 
 
 
 
 
 
(this mail will be sent only by once)
 
 
 
----------------------------------------
http://www.pandanic.com
 
--=_NextPart_2rfkindysadvnqw3nerasdf-- From loewis@informatik.hu-berlin.de Wed Oct 16 10:00:03 2002 From: loewis@informatik.hu-berlin.de (Martin v. =?iso-8859-1?q?L=F6wis?=) Date: 16 Oct 2002 11:00:03 +0200 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: <3DAD1099.1040507@lemburg.com> References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <3DAD1099.1040507@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > If you're suggesting a new API, why not have it return one of > the new datetime objects ? Wouldn't that require to implement this type first? Regards, Martin From mal@lemburg.com Wed Oct 16 10:40:25 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 16 Oct 2002 11:40:25 +0200 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <3DAD1099.1040507@lemburg.com> Message-ID: <3DAD3409.6060005@lemburg.com> Martin v. L=F6wis wrote: > "M.-A. Lemburg" writes: >=20 >=20 >>If you're suggesting a new API, why not have it return one of >>the new datetime objects ? >=20 >=20 > Wouldn't that require to implement this type first? AFAIK, PythonLabs is working on this. Even though development seems to have stalled recently... See CVS:/python/nondist/sandbox/datetime/ --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mal@lemburg.com Wed Oct 16 10:46:22 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 16 Oct 2002 11:46:22 +0200 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! Message-ID: <3DAD356E.5080308@lemburg.com> I was wondering whether this is an oversight or intended. Buffer objects can certainly be compared to strings on a byte-by-byte basis, so the compare result looks like a (long standing) bug to me. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mwh@python.net Wed Oct 16 11:50:05 2002 From: mwh@python.net (Michael Hudson) Date: 16 Oct 2002 11:50:05 +0100 Subject: [Python-Dev] Memory size overflows In-Reply-To: Greg Ewing's message of "Wed, 16 Oct 2002 18:13:04 +1300 (NZDT)" References: <200210160513.g9G5D4003141@oma.cosc.canterbury.ac.nz> Message-ID: <2mof9uslnm.fsf@starship.python.net> Greg Ewing writes: > Tim Peters : > > > I'll repeat that it's hard to do this correctly and quickly (c.f. earlier > > comments about the history of Python's subtly buggy attempts). > > The really tragic thing about all this is that most > machines have perfectly good facilities for detecting > overflow at the machine level, but no commonly used > HLL that I know of provides access to them, for some > unfathomable reason. Well, if we go that route we can do floating point properly too. An impression I've got is that because noone actually uses the overflow detection stuff, chip designers are placing less empahasis on implementing them efficiently. A particularly unpleasant circle :-( Cheers, M. -- If comp.lang.lisp *is* what vendors are relying on to make or break Lisp sales, that's more likely the problem than is the effect of any one of us on such a flimsy marketing strategy... -- Kent M Pitman, comp.lang.lisp From mwh@python.net Wed Oct 16 11:51:39 2002 From: mwh@python.net (Michael Hudson) Date: 16 Oct 2002 11:51:39 +0100 Subject: [Python-Dev] Memory size overflows In-Reply-To: Armin Rigo's message of "Wed, 16 Oct 2002 10:11:18 +0200 (CEST)" References: Message-ID: <2mlm4ysll0.fsf@starship.python.net> Armin Rigo writes: > True as well... Why are C macros so limited?... To remind us all that the world is a harsh place. Now lets rewrite Python in common lisp. Or pyrex. Cheers, M. -- I'm okay with intellegent buildings, I'm okay with non-sentient buildings. I have serious reservations about stupid buildings. -- Dan Sheppard, ucam.chat (from Owen Dunn's summary of the year) From guido@python.org Wed Oct 16 13:25:46 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 08:25:46 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: Your message of "Wed, 16 Oct 2002 08:08:29 +0200." References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> > > I just found a place in my own code that broke because stat now > > returns floats for mtime etc. I was using a Zope "OIBTree" object, > > which is a dict-like extension type mapping strings to ints; it > > very reasonbly doesn't like it if you assign a float to it. > > In this case, could you fix your code by truncating/rounding, or by > using a different container? I added int() around the st_mtime reference, because there's no "OFBTree" data type, and because second resolution was good enough. > > Maybe we can introduce a variant of the stat() function that returns > > floats, or alternative field names that are only available when using > > attributes (not when using the tuple-ish API)? > > Assuming that in all cases of breakage it would be possible to easily > "fix" the code somehow, I'd like to have a migration plan, following > PEP 5: I.e. together with adding a new feature for float time stamps, > we should deprecate the old feature (of int time stamps) - perhaps not > right now, but as a pending deprecation. Then, after some time, we > should drop the old feature and continue with just the new one. > > So I'd like to see a solution which does not look ugly when the > deprecated feature is gone and just the new feature stays. > > Perhaps a flag "times_are_float" would be the right approach? > N: flag defaults to False, users need to ask for floats explicitly > N+1: omitting the flag is deprecated > N+2: flag defaults to True, users need to ask for ints explicitly > N+5: the flag is removed > > There are variations to this plan, adding a phase where it is an error > to omit the flag, or where a pending deprecation is added. Agreed, but I don't like adding and then deleting a "temporary" new argument to stat() and fstat(). In this particular case, I think having a global flag should be good enough. We can easily rig the standard library not to depend on its value. It should be settable by calling a function, e.g. os.stat_times(int) or os.stat_times(float) If you think passing the actual type objects is too cute, we can call it stat_float_times() and pass a bool. Without an argument it could return the current status, with the twist that if the platform doesn't support float times it will always return int. Or maybe we can make it return the precision. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 16 13:29:29 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 08:29:29 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: Your message of "Wed, 16 Oct 2002 09:09:13 +0200." <3DAD1099.1040507@lemburg.com> References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <3DAD1099.1040507@lemburg.com> Message-ID: <200210161229.g9GCTTS21887@pcp02138704pcs.reston01.va.comcast.net> > If you're suggesting a new API, why not have it return one of > the new datetime objects ? That would require much more changes to the user code to use the new feature. And in many cases a basic timestamp is just what you need -- all you usually do is comparing them, occasionally printing them, and both are done just as easily using the int or float form. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 16 13:39:05 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 08:39:05 -0400 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: Your message of "Wed, 16 Oct 2002 11:46:22 +0200." <3DAD356E.5080308@lemburg.com> References: <3DAD356E.5080308@lemburg.com> Message-ID: <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> > I was wondering whether this is an oversight or intended. Buffer > objects can certainly be compared to strings on a byte-by-byte > basis, so the compare result looks like a (long standing) > bug to me. I'd consider it a feature, designed to convey the subliminal message "the buffer type should be deprecated". :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Wed Oct 16 14:04:48 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 16 Oct 2002 15:04:48 +0200 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3DAD63F0.4080409@lemburg.com> Guido van Rossum wrote: >>I was wondering whether this is an oversight or intended. Buffer >>objects can certainly be compared to strings on a byte-by-byte >>basis, so the compare result looks like a (long standing) >>bug to me. > > > I'd consider it a feature, designed to convey the subliminal message > "the buffer type should be deprecated". :-) Fine, but what alternative is there which meets the following requirements: * signals "this data is binary data" * compares just fine to strings * gets accepted by all APIs which use the buffer interface to access the data * has a C API which can be used in extensions * is available in Python 2.1.x and up (other than rolling my own mxBinary type...) What I'd basically need is a type which simply wraps up any string data, plays nice with 8-bit strings and signals the binary nature of the content. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From barry@python.org Wed Oct 16 14:06:16 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 16 Oct 2002 09:06:16 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15789.25672.851902.133899@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> In this particular case, I think having a global flag should GvR> be good enough. We can easily rig the standard library not GvR> to depend on its value. It should be settable by calling a GvR> function, e.g. GvR> os.stat_times(int) GvR> or GvR> os.stat_times(float) GvR> If you think passing the actual type objects is too cute, we GvR> can call it stat_float_times() and pass a bool. So then why not just make this an optional argument flag on the os.stat() call itself? That way you could migrate some code at a different rate than other code (a global flag might be fine for your application, but break some 3rd party library you're using). -Barry From barry@python.org Wed Oct 16 14:07:23 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 16 Oct 2002 09:07:23 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <15788.51819.500992.678200@gargle.gargle.HOWL> Message-ID: <15789.25739.530553.355804@gargle.gargle.HOWL> >>>>> "MvL" == Martin v Loewis writes: >> +1 on the latter idea. Since the st_mtime, st_atime, and >> st_ctime attributes, and the tuple-ish API are documented as >> returning ints, I think you shouldn't change that. Provide >> different field names for float values, e.g. f_mtime, f_ctime, >> f_atime, or maybe just mtime, atime, and ctime... ? MvL> The previous suggestion was st_mtimef, st_ctimef, ... Names which differ by one character at the end of the name are error prone, IME. Plus it makes stuff like dabbrev-expand harder to use. :) -Barry From martin@v.loewis.de Wed Oct 16 14:15:55 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 16 Oct 2002 15:15:55 +0200 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > In this particular case, I think having a global flag should be good > enough. We can easily rig the standard library not to depend on its > value. Sounds good. Unless I hear otherwise, I'll add stat_float_times(bool). That still leaves the question of a migration plan, assuming that users, in general, would appreciate to silently get subsecond timestamps where possible. Here is a proposal: - 2.3: just add stat_float_times. In the documentation, warn that the value will change in the future. - 2.4: make stat_float_times(True) the default. Applications that break would need to add stat_float_times(False) in a central place. This has no phase were a warning is emitted, but any scheme that adds a warning means that the user first has to add something (to silence the warning), and then to remove something (when the default changes - although the option might stay forever). Regards, Martin From martin@v.loewis.de Wed Oct 16 14:30:40 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 16 Oct 2002 15:30:40 +0200 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: <3DAD63F0.4080409@lemburg.com> References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > * signals "this data is binary data" > * compares just fine to strings > * gets accepted by all APIs which use the buffer interface to > access the data > * has a C API which can be used in extensions > * is available in Python 2.1.x and up I believe the string type meets all these requirements. Regards, Martin From jacobs@penguin.theopalgroup.com Wed Oct 16 14:44:27 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Wed, 16 Oct 2002 09:44:27 -0400 (EDT) Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: Message-ID: On 16 Oct 2002, Martin v. Loewis wrote: > Guido van Rossum writes: > > > In this particular case, I think having a global flag should be good > > enough. We can easily rig the standard library not to depend on its > > value. > > Sounds good. Unless I hear otherwise, I'll add stat_float_times(bool). Please don't! The global setting isn't sufficient since many of us access many 3rd party modules. Why not simply create properies that return the int/float versions from the same structure and store the float versions? That seems to be an instant win for all use-cases. My 2e-2cents, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From martin@v.loewis.de Wed Oct 16 15:09:10 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 16 Oct 2002 16:09:10 +0200 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: References: Message-ID: Kevin Jacobs writes: > > Sounds good. Unless I hear otherwise, I'll add stat_float_times(bool). > > Please don't! The global setting isn't sufficient since many of us access > many 3rd party modules. Why is that insufficient? As long as you have modules that require int time stamps, you can't switch to float time stamps. If you find you are using such a module, tell the module author about that, and wait for a corrected module. > Why not simply create properies that return the int/float versions > from the same structure and store the float versions? How can the property know whether to return an int or a float? Regards, Martin From mal@lemburg.com Wed Oct 16 15:12:13 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 16 Oct 2002 16:12:13 +0200 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> Message-ID: <3DAD73BD.2020805@lemburg.com> Martin v. Loewis wrote: > "M.-A. Lemburg" writes: > > >>* signals "this data is binary data" >>* compares just fine to strings >>* gets accepted by all APIs which use the buffer interface to >> access the data >>* has a C API which can be used in extensions >>* is available in Python 2.1.x and up > > > I believe the string type meets all these requirements. Except one which was implicit: how to tell binary data from text data. This information can sometimes be deduced from the string content provided you know what text data means to you, but this doesn't always work, since sometimes binary data happens to look like text data (ie. use only character ordinals as data bytes). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Wed Oct 16 15:20:57 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 10:20:57 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: Your message of "Wed, 16 Oct 2002 15:15:55 +0200." References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210161420.g9GEKvD01247@odiug.zope.com> > > In this particular case, I think having a global flag should be good > > enough. We can easily rig the standard library not to depend on its > > value. > > Sounds good. Unless I hear otherwise, I'll add stat_float_times(bool). > > That still leaves the question of a migration plan, assuming that > users, in general, would appreciate to silently get subsecond > timestamps where possible. Here is a proposal: > > - 2.3: just add stat_float_times. In the documentation, warn that the > value will change in the future. > - 2.4: make stat_float_times(True) the default. Applications that break > would need to add stat_float_times(False) in a central place. > > This has no phase were a warning is emitted, but any scheme that adds > a warning means that the user first has to add something (to silence > the warning), and then to remove something (when the default changes - > although the option might stay forever). Sounds good. I wonder if we could let the tuple-ish interface return ints forever (or until it's no longer supported). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 16 15:28:52 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 10:28:52 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: Your message of "Wed, 16 Oct 2002 09:06:16 EDT." <15789.25672.851902.133899@gargle.gargle.HOWL> References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> <15789.25672.851902.133899@gargle.gargle.HOWL> Message-ID: <200210161428.g9GESqE01333@odiug.zope.com> Barry: > So then why not just make this an optional argument flag on the > os.stat() call itself? That way you could migrate some code at a > different rate than other code (a global flag might be fine for your > application, but break some 3rd party library you're using). And Kevin: > Please don't! The global setting isn't sufficient since many of us > access many 3rd party modules. Most code doesn't care; most 3rd party code will work either way. Martin proposed adding an option to the stat() call itself first, but that seems a rather heavy-handed way to ensure backwards compatibility for such a very rare situation. If you are using a 3rd party library that breaks due to the floats, you can disable the floats until you've got time to fix it. You can live with the ints a little longer. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 16 15:31:02 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 10:31:02 -0400 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: Your message of "Wed, 16 Oct 2002 16:12:13 +0200." <3DAD73BD.2020805@lemburg.com> References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> <3DAD73BD.2020805@lemburg.com> Message-ID: <200210161431.g9GEV2R01348@odiug.zope.com> > >>* signals "this data is binary data" > >>* compares just fine to strings > >>* gets accepted by all APIs which use the buffer interface to > >> access the data > >>* has a C API which can be used in extensions > >>* is available in Python 2.1.x and up > > > > > > I believe the string type meets all these requirements. > > Except one which was implicit: how to tell binary data from > text data. This information can sometimes be deduced > from the string content provided you know what text data > means to you, but this doesn't always work, since sometimes > binary data happens to look like text data (ie. use only > character ordinals as data bytes). I don't understand why you need to signal "this is binary data" while at the same time you want to be able to compare to strings. Also, since buffer objects *can't* be compared to strings right now, and you require compatibility with 2.1, there is no solution that satisfies your requirements, so I conclude you're just being "difficult". :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Wed Oct 16 15:28:04 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 16 Oct 2002 16:28:04 +0200 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: <200210161420.g9GEKvD01247@odiug.zope.com> References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> <200210161420.g9GEKvD01247@odiug.zope.com> Message-ID: Guido van Rossum writes: > I wonder if we could let the tuple-ish interface return ints forever > (or until it's no longer supported). That would not be easy to implement. The structseq type does not support unnamed sequence members (yet), which we'ld need: the int members would have only indices, and the float members only names. It probably could be made work by allowing NULL names in a PyStructSequence_Field. I'll try to do this. Regards, Martin From martin@v.loewis.de Wed Oct 16 15:31:38 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 16 Oct 2002 16:31:38 +0200 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: <3DAD73BD.2020805@lemburg.com> References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> <3DAD73BD.2020805@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > Except one which was implicit: how to tell binary data from > text data. I see. In this case, class BinaryString(str): pass should do. The string type *is* the type for binary data. For character data, unicode is the type. Regards, Martin From guido@python.org Wed Oct 16 15:41:40 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 10:41:40 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: Your message of "Wed, 16 Oct 2002 16:28:04 +0200." References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> <200210161420.g9GEKvD01247@odiug.zope.com> Message-ID: <200210161441.g9GEfew01452@odiug.zope.com> > > I wonder if we could let the tuple-ish interface return ints forever > > (or until it's no longer supported). > > That would not be easy to implement. The structseq type does not > support unnamed sequence members (yet), which we'ld need: the int > members would have only indices, and the float members only names. > > It probably could be made work by allowing NULL names in a > PyStructSequence_Field. I'll try to do this. That sounds good to me. Or you could "subclass" the PyStructSequence class with something that special-cases the st_?time attributes. --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs@penguin.theopalgroup.com Wed Oct 16 15:50:14 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Wed, 16 Oct 2002 10:50:14 -0400 (EDT) Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: Message-ID: On 16 Oct 2002, Martin v. Loewis wrote: > Kevin Jacobs writes: > > > > Sounds good. Unless I hear otherwise, I'll add stat_float_times(bool). > > > > Please don't! The global setting isn't sufficient since many of us access > > many 3rd party modules. > > Why is that insufficient? As long as you have modules that require int > time stamps, you can't switch to float time stamps. If you find you > are using such a module, tell the module author about that, and wait > for a corrected module. Why break all modules when only one needs the old behavior? Think about how from __future__ import is set up. > > Why not simply create properies that return the int/float versions > > from the same structure and store the float versions? > > How can the property know whether to return an int or a float? By name -- anyone who wants a floating point value can request fst_mtime vs. st_mtime, or whatever. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From guido@python.org Wed Oct 16 16:01:19 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 11:01:19 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: Your message of "Wed, 16 Oct 2002 10:50:14 EDT." References: Message-ID: <200210161501.g9GF1JO01579@odiug.zope.com> > Why break all modules when only one needs the old behavior? Think > about how from __future__ import is set up. Because this is IMO a *very* small change, for which I don't want to invoke such heavy guns. --Guido van Rossum (home page: http://www.python.org/~guido/) From gsw@agere.com Wed Oct 16 16:12:05 2002 From: gsw@agere.com (Gerald S. Williams) Date: Wed, 16 Oct 2002 11:12:05 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: <20021016131701.12398.86665.Mailman@mail.python.org> Message-ID: Guido van Rossum wrote: > I.e. a macro callable as > SAFE_MULTIPLY(destination, src1, src2, on_overflow); > meaning roughly > destination = src1 * src2; > if () > on_overflow; > There are also additions...These are easier to test: if you add a small > positive constant to a size_t, and the result is smaller than the > original size_t, an overflow occurred. Why not use the same trick for multiplication? For src1,src2 > 0, dest should always be >= MAX(src1,src2). SAFE_MULTIPLY could be implemented something like this: #define HALF_BITS (sizeof(size_t) * 4U) #define LO_MASK ((((size_t)1) << (HALF_BITS))-1) #define HI_MASK (~(LO_MASK)) #define MAX(a,b) (((a) >= (b)) ? (a) : (b)) #define SAFE_MULTIPLY(dest,src1,src2,on_error) \ { \ size_t _x = src1; \ size_t _y = src2; \ size_t _dest = _x * _y; \ \ if (_x && _y && ((_x|_y) & HI_MASK)) \ { \ if (_dest < MAX(_x,_y)) \ { \ on_error; \ } \ } \ \ dest = _dest; \ } -Jerry Williams From barry@python.org Wed Oct 16 16:09:10 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 16 Oct 2002 11:09:10 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> <15789.25672.851902.133899@gargle.gargle.HOWL> <200210161428.g9GESqE01333@odiug.zope.com> Message-ID: <15789.33046.223487.447141@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> Most code doesn't care; most 3rd party code will work either GvR> way. Martin proposed adding an option to the stat() call GvR> itself first, but that seems a rather heavy-handed way to GvR> ensure backwards compatibility for such a very rare GvR> situation. GvR> If you are using a 3rd party library that breaks due to the GvR> floats, you can disable the floats until you've got time to GvR> fix it. You can live with the ints a little longer. But you may end up ping ponging between floats and ints as you import new libraries into your app. If, as you surmise, most code won't care, then that won't be a real problem. But if it is, you might see code like this: ostate = os.stat_times() os.stat_times(True) try: # do something requiring float times finally: os.stat_times(ostate) Maybe it won't matter in practice, but global state like that always tickles the gag reflex in me. ;) -Barry From martin@v.loewis.de Wed Oct 16 16:21:26 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 16 Oct 2002 17:21:26 +0200 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: References: Message-ID: Kevin Jacobs writes: > Why break all modules when only one needs the old behavior? Think about how > from __future__ import is set up. Nothing will break; that's the entire purpose of this change. > By name -- anyone who wants a floating point value can request > fst_mtime vs. st_mtime, or whatever. Been there, done that (posixmodule.c 2.254). Please understand that, after the transitional phase, fst_mtime is what everyone will use. This is plain ugly, since it is inconsistent with the names of all other fields. So all we need to do is to accommodate the transition, and we do that with Guido's proposed change. Regards, Martin From mal@lemburg.com Wed Oct 16 16:24:20 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 16 Oct 2002 17:24:20 +0200 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> <3DAD73BD.2020805@lemburg.com> <200210161431.g9GEV2R01348@odiug.zope.com> Message-ID: <3DAD84A4.6020008@lemburg.com> Guido van Rossum wrote: >>>>* signals "this data is binary data" >>>>* compares just fine to strings >>>>* gets accepted by all APIs which use the buffer interface to >>>> access the data >>>>* has a C API which can be used in extensions >>>>* is available in Python 2.1.x and up >>> >>> >>>I believe the string type meets all these requirements. >> >>Except one which was implicit: how to tell binary data from >>text data. This information can sometimes be deduced >>from the string content provided you know what text data >>means to you, but this doesn't always work, since sometimes >>binary data happens to look like text data (ie. use only >>character ordinals as data bytes). > > I don't understand why you need to signal "this is binary data" while > at the same time you want to be able to compare to strings. Because I use buffer objects to wrap string data to say "this is binary data" to a database. When fetching the same data back from the database I return a string and I found the quirk mentioned in the subject while writing a unit test for this. It's not a showstopper. The above just was a hint not to deprecate the buffer object until we've come up with a decent replacement that's easy to adapt in existing code. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Wed Oct 16 16:25:19 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 16 Oct 2002 17:25:19 +0200 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: <15789.33046.223487.447141@gargle.gargle.HOWL> References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> <15789.25672.851902.133899@gargle.gargle.HOWL> <200210161428.g9GESqE01333@odiug.zope.com> <15789.33046.223487.447141@gargle.gargle.HOWL> Message-ID: barry@python.org (Barry A. Warsaw) writes: > But you may end up ping ponging between floats and ints as you import > new libraries into your app. If, as you surmise, most code won't > care, then that won't be a real problem. With Guido's most recent suggestion (put ints into the tuple), it is even more likely that nothing will break if floats are activated (we should investigate the two known incidents to find out whether they were using the tuple interface). > # do something requiring float times I think this is a central point here: Why would anything *require* float times? If the system does not support subsecond timestamps, or Python does not expose that (e.g. Windows), you will get floats, but you won't get fractions of a second. > Maybe it won't matter in practice, but global state like that always > tickles the gag reflex in me. ;) I completely agree about reflexes and general feelings. However, we assume that this indeed won't matter in practice. Regards, Martin From jepler@unpythonic.net Wed Oct 16 16:26:10 2002 From: jepler@unpythonic.net (Jeff Epler) Date: Wed, 16 Oct 2002 10:26:10 -0500 Subject: [Python-Dev] Memory size overflows In-Reply-To: References: <20021016131701.12398.86665.Mailman@mail.python.org> Message-ID: <20021016152609.GC9288@unpythonic.net> On Wed, Oct 16, 2002 at 11:12:05AM -0400, Gerald S. Williams wrote: > Guido van Rossum wrote: > > I.e. a macro callable as > > SAFE_MULTIPLY(destination, src1, src2, on_overflow); > > meaning roughly > > destination = src1 * src2; > > if () > > on_overflow; > > > There are also additions...These are easier to test: if you add a small > > positive constant to a size_t, and the result is smaller than the > > original size_t, an overflow occurred. > > Why not use the same trick for multiplication? For src1,src2 > 0, > dest should always be >= MAX(src1,src2). SAFE_MULTIPLY could be > implemented something like this: [...] Is this correct for all operands? Consider 4-bit types, with valid values 0..15. 3*9 == 27, but 27%16 == 11 and 11>max(3, 9) There seem to be 15 distinct operand pairs for which this fails in the 4-bit case. def check(m): for x in range(m): for y in range(x, m): z = x*y w = z % m if z != w and w>x and w>y: yield x, y, z, w >>> print list(check(16)) [(3, 9, 27, 11), (3, 10, 30, 14), (4, 6, 24, 8), (4, 7, 28, 12), (4, 11, 44, 12), (5, 5, 25, 9), (5, 6, 30, 14), (5, 9, 45, 13), (6, 7, 42, 10), (6, 10, 60, 12), (6, 13, 78, 14), (7, 9, 63, 15), (7, 11, 77, 13), (10, 11, 110, 14), (11, 13, 143, 15)] >>> print len(list(check(256)) 9915 Jeff From guido@python.org Wed Oct 16 16:36:29 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 11:36:29 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: Your message of "Wed, 16 Oct 2002 11:09:10 EDT." <15789.33046.223487.447141@gargle.gargle.HOWL> References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> <15789.25672.851902.133899@gargle.gargle.HOWL> <200210161428.g9GESqE01333@odiug.zope.com> <15789.33046.223487.447141@gargle.gargle.HOWL> Message-ID: <200210161536.g9GFaTI01841@odiug.zope.com> > Maybe it won't matter in practice, but global state like that always > tickles the gag reflex in me. ;) Believe me, mine too -- if I thought it was a real problem. But I don't. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Wed Oct 16 16:31:02 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 16 Oct 2002 11:31:02 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> <15789.25672.851902.133899@gargle.gargle.HOWL> <200210161428.g9GESqE01333@odiug.zope.com> <15789.33046.223487.447141@gargle.gargle.HOWL> Message-ID: <15789.34358.965278.784246@gargle.gargle.HOWL> >>>>> "MvL" == Martin v Loewis writes: >> But you may end up ping ponging between floats and ints as you >> import new libraries into your app. If, as you surmise, most >> code won't care, then that won't be a real problem. MvL> With Guido's most recent suggestion (put ints into the MvL> tuple), it is even more likely that nothing will break if MvL> floats are activated (we should investigate the two known MvL> incidents to find out whether they were using the tuple MvL> interface). Cool, that sounds like the best compromise. -Barry From guido@python.org Wed Oct 16 16:40:25 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 11:40:25 -0400 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: Your message of "Wed, 16 Oct 2002 17:24:20 +0200." <3DAD84A4.6020008@lemburg.com> References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> <3DAD73BD.2020805@lemburg.com> <200210161431.g9GEV2R01348@odiug.zope.com> <3DAD84A4.6020008@lemburg.com> Message-ID: <200210161540.g9GFePD01901@odiug.zope.com> > The above just was a hint not to deprecate the buffer > object until we've come up with a decent replacement > that's easy to adapt in existing code. So, apart from the Python 2.1 requirement, subclassing str does the trick, right? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 16 16:41:41 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 11:41:41 -0400 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: Your message of "Wed, 16 Oct 2002 17:25:19 +0200." References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <200210161225.g9GCPkc21859@pcp02138704pcs.reston01.va.comcast.net> <15789.25672.851902.133899@gargle.gargle.HOWL> <200210161428.g9GESqE01333@odiug.zope.com> <15789.33046.223487.447141@gargle.gargle.HOWL> Message-ID: <200210161541.g9GFffv01914@odiug.zope.com> > With Guido's most recent suggestion (put ints into the tuple), it is > even more likely that nothing will break if floats are activated (we > should investigate the two known incidents to find out whether they > were using the tuple interface). I was using the tuple interface -- this code needed to work under Python 2.1. > I completely agree about reflexes and general feelings. However, we > assume that this indeed won't matter in practice. Amen. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Wed Oct 16 16:45:02 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 16 Oct 2002 17:45:02 +0200 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> <3DAD73BD.2020805@lemburg.com> <200210161431.g9GEV2R01348@odiug.zope.com> <3DAD84A4.6020008@lemburg.com> <200210161540.g9GFePD01901@odiug.zope.com> Message-ID: <3DAD897E.6060800@lemburg.com> Guido van Rossum wrote: >>The above just was a hint not to deprecate the buffer >>object until we've come up with a decent replacement >>that's easy to adapt in existing code. > > > So, apart from the Python 2.1 requirement, subclassing str does the > trick, right? Right. Would be nice if there were a standard builtin, e.g. binary(), for this and maybe some support code to go with it in C (e.g. the type object would be nice to have at C level). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Wed Oct 16 17:01:47 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 12:01:47 -0400 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: Your message of "Wed, 16 Oct 2002 17:45:02 +0200." <3DAD897E.6060800@lemburg.com> References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> <3DAD73BD.2020805@lemburg.com> <200210161431.g9GEV2R01348@odiug.zope.com> <3DAD84A4.6020008@lemburg.com> <200210161540.g9GFePD01901@odiug.zope.com> <3DAD897E.6060800@lemburg.com> Message-ID: <200210161601.g9GG1mM02180@odiug.zope.com> > > So, apart from the Python 2.1 requirement, subclassing str does the > > trick, right? > > Right. > > Would be nice if there were a standard builtin, e.g. binary(), > for this and maybe some support code to go with it in C (e.g. > the type object would be nice to have at C level). I disagree. There are a thousand different applications, and yours seems rather unusual to me. --Guido van Rossum (home page: http://www.python.org/~guido/) From gsw@agere.com Wed Oct 16 17:00:06 2002 From: gsw@agere.com (Gerald S. Williams) Date: Wed, 16 Oct 2002 12:00:06 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: <20021016152609.GC9288@unpythonic.net> Message-ID: Jeff Epler wrote: > Is this correct for all operands? Consider 4-bit types, with valid > values 0..15. > 3*9 == 27, but 27%16 == 11 and 11>max(3, 9) You're right. It's a case of "fingers quicker than the brain". There may be something worth investigating in that type of relationship, but that isn't the answer. I came up with it as I was responding with a somewhat more working answer. What I was originally going to post before my mental lapse was this: One solution to what Guido asked for is as follows. It avoids the division, but I don't think it's worth it (it doesn't make much difference on my platform): #define FULL_BITS (sizeof(size_t) * 8U) #define TOP_BIT (((size_t)1) << ((FULL_BITS)-1)) #define HALF_BITS (sizeof(size_t) * 4U) #define LO_MASK ((((size_t)1) << (HALF_BITS))-1) #define HI_MASK (~(LO_MASK)) #define SAFE_MULTIPLY(dest,src1,src2,on_error)\ {\ size_t _x = src1;\ size_t _y = src2;\ \ dest = _x * _y;\ \ if ((_x|_y) & HI_MASK)\ {\ size_t h;\ size_t l;\ size_t mask;\ size_t low_bit;\ size_t low_bit_mask;\ \ if (_x >= _y)\ {\ h = _x;\ l = _y;\ }\ else\ {\ h = _y;\ l = _x;\ }\ \ for (low_bit=0,mask=TOP_BIT; mask & HI_MASK; ++low_bit,(mask>>=1))\ {\ if (h & mask)\ {\ break;\ }\ }\ \ low_bit_mask = (size_t)1 << low_bit;\ \ do\ {\ if (l & mask)\ {\ on_error;\ }\ mask >>= 1;\ }\ while (!(mask & low_bit_mask));\ }\ } Well, at least it demonstrates how to avoid multiply-invoking its parameters. :-) -Jerry From mal@lemburg.com Wed Oct 16 17:15:02 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 16 Oct 2002 18:15:02 +0200 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> <3DAD73BD.2020805@lemburg.com> <200210161431.g9GEV2R01348@odiug.zope.com> <3DAD84A4.6020008@lemburg.com> <200210161540.g9GFePD01901@odiug.zope.com> <3DAD897E.6060800@lemburg.com> <200210161601.g9GG1mM02180@odiug.zope.com> Message-ID: <3DAD9086.1040204@lemburg.com> Guido van Rossum wrote: >>>So, apart from the Python 2.1 requirement, subclassing str does the >>>trick, right? >> >>Right. >> >>Would be nice if there were a standard builtin, e.g. binary(), >>for this and maybe some support code to go with it in C (e.g. >>the type object would be nice to have at C level). > > > I disagree. There are a thousand different applications, and yours > seems rather unusual to me. It's not at all unusual if you interface to databases. These offer you three choices: character data, Unicode data and binary data and each of these is handled slightly differently. We currently don't have any notion of separating character data from binary except the difference between Unicode and strings. Using Unicode for character data only and reserving strings for binary data would be nice, except that practice shows that this doesn't always work because not all tools in the chain are ready for Unicode just yet (including Python's stdlib itself). Nevermind, I'll roll my own, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Wed Oct 16 17:26:59 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 12:26:59 -0400 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: Your message of "Wed, 16 Oct 2002 18:15:02 +0200." <3DAD9086.1040204@lemburg.com> References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> <3DAD73BD.2020805@lemburg.com> <200210161431.g9GEV2R01348@odiug.zope.com> <3DAD84A4.6020008@lemburg.com> <200210161540.g9GFePD01901@odiug.zope.com> <3DAD897E.6060800@lemburg.com> <200210161601.g9GG1mM02180@odiug.zope.com> <3DAD9086.1040204@lemburg.com> Message-ID: <200210161626.g9GGQxM08939@odiug.zope.com> > >>Would be nice if there were a standard builtin, e.g. binary(), > >>for this and maybe some support code to go with it in C (e.g. > >>the type object would be nice to have at C level). > > > > I disagree. There are a thousand different applications, and yours > > seems rather unusual to me. > > It's not at all unusual if you interface to databases. These offer > you three choices: character data, Unicode data and binary data > and each of these is handled slightly differently. I figure that most apps will be happy to return 8-bit strings for binary data; that's what 8-bit strings were explicitly designed to support. > We currently don't have any notion of separating character data from > binary except the difference between Unicode and strings. Using > Unicode for character data only and reserving strings for > binary data would be nice, except that practice shows that this > doesn't always work because not all tools in the chain are > ready for Unicode just yet (including Python's stdlib itself). No, we'll eventually need a separate data type, but I doubt it needs to be as compatible with the current 8-bit string type as your requirements state. > Nevermind, I'll roll my own, You're welcome. --Guido van Rossum (home page: http://www.python.org/~guido/) From xscottg@yahoo.com Wed Oct 16 17:27:23 2002 From: xscottg@yahoo.com (Scott Gilbert) Date: Wed, 16 Oct 2002 09:27:23 -0700 (PDT) Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: <3DAD9086.1040204@lemburg.com> Message-ID: <20021016162723.13099.qmail@web40102.mail.yahoo.com> --- "M.-A. Lemburg" wrote: > > We currently don't have any notion of separating character data from > binary except the difference between Unicode and strings. Using > Unicode for character data only and reserving strings for > binary data would be nice, except that practice shows that this > doesn't always work because not all tools in the chain are > ready for Unicode just yet (including Python's stdlib itself). > > Nevermind, I'll roll my own, > I don't know if you've seen PEP 296, but I still hope to finish it and have it accepted in time for the Python 2.3 release. That doesn't meet your requirement of working with Python 2.1 however. __________________________________________________ Do you Yahoo!? Faith Hill - Exclusive Performances, Videos & More http://faith.yahoo.com From tim.one@comcast.net Wed Oct 16 17:31:44 2002 From: tim.one@comcast.net (Tim Peters) Date: Wed, 16 Oct 2002 12:31:44 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: <20021016081532.GA49294@hishome.net> Message-ID: [Tim] > BTW, if we know we're not interested in anything other than > 32-bit products, > > double _prod = (double)src1 * (double)src2; > if (_prod < 4294967296.0) /* 2**32 */ > result = (size_t)_prod; > else > overflow; [Oren Tirosh] > s/_prod;/src1 * src2;/ > > Re-calculating the product with integer multiplication is probably faster > than converting a double to integer on any sane CPU. This is so. Better, because it allows more instruction-level parallelism: double dprod = (double)src1 * (double)src2; size_t iprod = src1 * src2; if (dprod < 4294967296.0) /* 2**32 */ result = iprod; else /* overflow */ > I wonder if the old SPARCs that don't have an integer multiply > instruction actually have a floating point to integer conversion > that is faster than integer mul. I'll let someone with an old SPARC worry about that. > Wasting "expensive" multiplications this way still feels blasphemous :) Well, we could simulate multiplication by hand, with a bit-at-a-time shift-and-add loop. This reduces the detect-mul-overflow case to the detect-add-overflow case. Bonus: on some odd boxes, it would be faster . From tim.one@comcast.net Wed Oct 16 17:37:54 2002 From: tim.one@comcast.net (Tim Peters) Date: Wed, 16 Oct 2002 12:37:54 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: [Guido] > The trick will to ensure that the macro arguments are used exactly > once in all versions (otherwise switching macros might unveal a new > class of> > bugs). [Armin Rigo] > Harder... It's trivial if you define the macro to take an lvalue result argument. Then the macro can expand to a block, and so declare block-local temp variables, including, if desired, temps that are merely initialized to other macro arguments. Such temps can be reused as often as you like without semantic consquence. Contorting this into expression rvalue form is a challenge, but a worthier challenge is yielding to common sense here . From martin@v.loewis.de Wed Oct 16 18:11:21 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 16 Oct 2002 19:11:21 +0200 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: <200210161601.g9GG1mM02180@odiug.zope.com> References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> <3DAD73BD.2020805@lemburg.com> <200210161431.g9GEV2R01348@odiug.zope.com> <3DAD84A4.6020008@lemburg.com> <200210161540.g9GFePD01901@odiug.zope.com> <3DAD897E.6060800@lemburg.com> <200210161601.g9GG1mM02180@odiug.zope.com> Message-ID: Guido van Rossum writes: > > Would be nice if there were a standard builtin, e.g. binary(), > > for this and maybe some support code to go with it in C (e.g. > > the type object would be nice to have at C level). > > I disagree. There are a thousand different applications, and yours > seems rather unusual to me. I do think there should be a string type for binary data, and that the standard string type should become Unicode one day. There was past discussion about this, at which proponents suggest that there should be even binary literals. Of course, all I/O would use the binary, unless an encoding was specified when creating the stream. Regards, Martin From guido@python.org Wed Oct 16 18:19:07 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 16 Oct 2002 13:19:07 -0400 Subject: [Python-Dev] buffer('abc') == 'abc' is False ?! In-Reply-To: Your message of "Wed, 16 Oct 2002 19:11:21 +0200." References: <3DAD356E.5080308@lemburg.com> <200210161239.g9GCd5v22010@pcp02138704pcs.reston01.va.comcast.net> <3DAD63F0.4080409@lemburg.com> <3DAD73BD.2020805@lemburg.com> <200210161431.g9GEV2R01348@odiug.zope.com> <3DAD84A4.6020008@lemburg.com> <200210161540.g9GFePD01901@odiug.zope.com> <3DAD897E.6060800@lemburg.com> <200210161601.g9GG1mM02180@odiug.zope.com> Message-ID: <200210161719.g9GHJ7E09833@odiug.zope.com> > I do think there should be a string type for binary data, and that the > standard string type should become Unicode one day. There was past > discussion about this, at which proponents suggest that there should > be even binary literals. > > Of course, all I/O would use the binary, unless an encoding was > specified when creating the stream. Maybe it's time for a PEP outlining and detailing this view of the future? --Guido van Rossum (home page: http://www.python.org/~guido/) From gsw@agere.com Wed Oct 16 21:30:55 2002 From: gsw@agere.com (Gerald S. Williams) Date: Wed, 16 Oct 2002 16:30:55 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: Actually, my original solution doesn't seem to stand up to comprehensive testing, either. Essentially all it was doing was checking to see if the sum of the high-order bit positions was less than the number of bits. I created a Python equivalent for smaller width numbers. At a width of 6 bits, the following combinations aren't reported as overflows but should be: (6, 13) (6, 14) (6, 15) (7, 11) (7, 12) (7, 13) (7, 14) (7, 15) Perhaps there is some property of the false negatives that can be exploited. I don't know if it helps, but there are certainly a number of ways to narrow the field: if (src1 & HIMASK) && (src2 & HIMASK), overflow=True if !(src1 & HIMASK) && !(src2 & HIMASK), overflow=False if !src1 || !src2, overflow=False if dest < MAX(src1,src2), overflow=True if log2(src1) + log2(src2) >= width, overflow=True (the integer form of log2 can be done quite cheaply by exploiting known properties of src1 and src2 at this point) But you'd still need to filter out false negatives from that field. Perhaps there's a way, but divides would have to be really expensive for this to be worth it. -Jerry From greg@cosc.canterbury.ac.nz Wed Oct 16 22:32:48 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Oct 2002 10:32:48 +1300 (NZDT) Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: Message-ID: <200210162132.g9GLWmX09153@oma.cosc.canterbury.ac.nz> martin@v.loewis.de (Martin v. Loewis): > The previous suggestion was st_mtimef, st_ctimef, ... For some reason, st_fmtime and st_fctime seem easier to read and pronounce to me. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg@cosc.canterbury.ac.nz Wed Oct 16 22:39:23 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Oct 2002 10:39:23 +1300 (NZDT) Subject: [Python-Dev] Memory size overflows In-Reply-To: <20021016081532.GA49294@hishome.net> Message-ID: <200210162139.g9GLdNQ09171@oma.cosc.canterbury.ac.nz> > Re-calculating the product with integer multiplication is probably faster > than converting a double to integer on any sane CPU. Um... what would be "sane" about that? I don't know much about the Pentium. I do know that the PowerPC doesn't have any way of moving a value from a float to an integer register without going through memory... but I regard that as a decidedly INsane feature! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From gsw@agere.com Thu Oct 17 05:36:23 2002 From: gsw@agere.com (Gerald S. Williams) Date: Thu, 17 Oct 2002 00:36:23 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: I felt obliged to post a working copy of SAFE_MULTIPLY. On my platform, it's slower than simply dividing by one of the multiplicands and comparing to the other, but it does get the job done without division. This is what I started out to write before my brain short-circuited. Here's a simplified Python representation of the algorithm: def check(x,y,m): xrem, xlog = math.frexp(x) yrem, ylog = math.frexp(y) if xlog + ylog == m + 1: return (xrem * yrem >= 0.5) else: return xlog + ylog > m + 1 And here it is in C: #define FULL_BITS (sizeof(size_t) * 8U) #define TOP_BIT (((size_t)1) << ((FULL_BITS)-1)) #define HALF_BITS (sizeof(size_t) * 4U) #define SAFE_MULTIPLY(dest,src1,src2,on_overflow)\ {\ size_t _x = src1;\ size_t _y = src2;\ size_t _dest = _x * _y;\ size_t h;\ int hlog;\ size_t l;\ int llog;\ size_t mask;\ \ dest = _dest;\ \ if (_x >= _y)\ {\ h = _x;\ l = _y;\ }\ else\ {\ h = _y;\ l = _x;\ }\ \ if (l)\ {\ for ((mask=TOP_BIT),(hlog=FULL_BITS-1); !(mask & h); mask >>= 1)\ {\ --hlog;\ }\ if (hlog >= HALF_BITS)\ {\ for (llog = hlog; !(mask & l); mask >>= 1)\ {\ --llog;\ }\ if (((hlog + llog) > (FULL_BITS - 1)) ||\ (((hlog + llog) == (FULL_BITS - 1)) &&\ (((h >> 1) * l + ((h & 1) ? (l >> 1) : 0)) >= TOP_BIT)))\ {\ on_overflow;\ }\ }\ }\ } -Jerry From tismer@tismer.com Thu Oct 17 07:03:43 2002 From: tismer@tismer.com (Christian Tismer) Date: Wed, 16 Oct 2002 23:03:43 -0700 Subject: [Python-Dev] Ann: Stackless technology merge under the way Message-ID: <3DAE52BF.9040304@tismer.com> Hi friends, Stackless is changing heavily, once again. After yesterday's move to Python 2.2.2, I am now working on an integration of the old technology (software only) and the new (modifying the hardware stack). This has a number of advantages: - porting of existing applications, which depend on the old changes to Python's interpreter, becomes much easier. They tend to rely on a Python version of 2.0 or lower and use the old structures at the same time. Supporting these, again, makes it possible to do stepwise upgrades. - existing applications like thread pickling (written by TwinSun.com) will be enabled again and merged into Stackless as a feature. While limited to plain frame contexts (no C function or method calls in between), this is very worth doing. - the core Stackless code is using C stack manipulations right now in order to reduce stack size. This is not guaranteed to work with every C extension, since it is hard to check whether a foreign C call is sitting on the stack and relying on variables which must stay in place. The old technology was only able to switch Python stacks, when no recursive C calls were existing. This locks switches of C extensions automagically. - the new technology can be removed from the Python core and be only used for the stackless extension, for tasklet switching. This is not decided yet, but an option. - the amount of C stack space to me moved around is reduced drastically by this hybrid approach, since most of these vanish automatically. Although stack compression has been prepared (and will be supported), this is more time efficient. The reason while I have no longer a problem to make this decision is Python's current state. After generators have been introduced, the core code already has a lot of properties which I need to enable the old technology with ease. Furthermore, this support code will not again try to cover everything, but just the minimum to avoid most recursion. It is neither planned to support stackless map or apply, nor are continuations desired. The target are applications which implemented coroutines on top of the old kernel. This hybrid implementation has started today and will continue over a couple of weeks. The last "plain hardware switching" version is available under the CVS tag "last_hardware_only", which is based on the final 2.2.2 version. I regard this move as the beginning of the re-union of my brain halves :-) continuation-less - ly y'rs chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tino.lange@isg.de Thu Oct 17 09:38:39 2002 From: tino.lange@isg.de (Tino Lange) Date: Thu, 17 Oct 2002 10:38:39 +0200 Subject: [Python-Dev] PyEval_EvalCode() Message-ID: <3DAE770F.5030304@isg.de> Hi! This prototype is not available when including "Python.h" (Using v2.1.3) Why did you exclude it from the "official" API prototypes? I found it in "eval.h", which has to be included manually, besides "Python.h". Isn't that PyEval_EvalCode() the regular API function to run CodeObjects from Py_CompileString() ? Cheers, Tino From arigo@tunes.org Thu Oct 17 11:57:33 2002 From: arigo@tunes.org (Armin Rigo) Date: Thu, 17 Oct 2002 12:57:33 +0200 (CEST) Subject: [Python-Dev] Ann: Stackless technology merge under the way In-Reply-To: <3DAE52BF.9040304@tismer.com> Message-ID: Hello Christian, On Wed, 16 Oct 2002, Christian Tismer wrote: > Stackless is changing heavily, once again. Good news. It might be worth pointing out that even if continuations are not directly supported, it should be possible to do them with thread pickling. Unpickling a given thread state several times does the trick. Armin From jason@tishler.net Thu Oct 17 13:18:16 2002 From: jason@tishler.net (Jason Tishler) Date: Thu, 17 Oct 2002 08:18:16 -0400 Subject: [Python-Dev] Dropping support for Tcl 8.0 and 8.1 In-Reply-To: References: <200210010607.g9167Mns001471@mira.informatik.hu-berlin.de> <200210011338.g91Dclo08817@odiug.zope.com> <200210011519.g91FJwI21585@odiug.zope.com> <20021015175131.GB1536@tishler.net> Message-ID: <20021017121816.GC2104@tishler.net> Martin, On Tue, Oct 15, 2002 at 10:57:58PM +0200, Martin v. Loewis wrote: > Jason Tishler writes: > > This is probably too little, too late, but Cygwin is still stuck at > > Tcl 8.0. Would you be willing to keep this support in? > > No, I've already made the changes to drop this support, and added > support for UCS-4 mode in _tkinter. Although the above is disappointing, your position is quite understandable. See below... > > Note that I've tried to encourage Cygwin to upgrade to a more recent > > version: > > > > http://cygwin.com/ml/cygwin/2002-10/msg00798.html > > That seems the right way to go. Agreed. > They may have changed Cygwin by the time Python 2.3 is released. Unfortunately, the above is unlikely. There seems to be "issues" with more recent Tcl versions under Cygwin. This implies that Cygwin Python 2.3+ will no longer provide a pre-built _tkinter module until the issues have been resolved. :,( > If not, users wishing to use Cygwin _tkinter can use Mumit Khan's Tcl > 8.3/8.4 ports: > > ftp://ftp.nanotech.wisc.edu/pub/khan/tcl/tcltk-8.3.4-cygwin/ The above appears to be the most likely workaround. Jason From gsw@agere.com Thu Oct 17 15:07:12 2002 From: gsw@agere.com (Gerald S. Williams) Date: Thu, 17 Oct 2002 10:07:12 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: Here's a version of my previous algorithm with the checks added back in to improve overall performance. This one runs about as fast as the divide-and-compare algorithm for my test cases. :-) OK, I'm done tweaking now... #define FULL_BITS (sizeof(size_t) * 8U) #define TOP_BIT (((size_t)1) << ((FULL_BITS)-1)) #define HALF_BITS (sizeof(size_t) * 4U) #define MID_BIT (((size_t)1) << ((HALF_BITS)-1)) #define LO_MASK ((((size_t)1) << (HALF_BITS))-1) #define HI_MASK (~(LO_MASK)) #define SAFE_MULTIPLY(dest,src1,src2,on_overflow)\ {\ size_t _x = src1;\ size_t _y = src2;\ size_t _dest = _x * _y;\ \ dest = _dest;\ \ if ((_x | _y) & HI_MASK)\ {\ size_t h;\ int hlog;\ size_t l;\ int llog;\ size_t mask;\ \ if (_x >= _y)\ {\ h = _x;\ l = _y;\ }\ else\ {\ h = _y;\ l = _x;\ }\ \ if ((h & HI_MASK) && (l))\ {\ if (l & HI_MASK)\ {\ hlog = llog = FULL_BITS;\ }\ else\ {\ for ((mask=TOP_BIT),(hlog=FULL_BITS-1);!(mask&h);mask>>=1)\ {\ --hlog;\ }\ for ((mask=MID_BIT),(llog=HALF_BITS-1);!(mask&l);mask>>=1)\ {\ --llog;\ }\ }\ if ((hlog + llog > FULL_BITS - 1) ||\ ((hlog + llog == FULL_BITS - 1) && !(_dest & TOP_BIT)))\ {\ on_overflow;\ }\ }\ }\ } -Jerry From arigo@tunes.org Thu Oct 17 16:16:42 2002 From: arigo@tunes.org (Armin Rigo) Date: Thu, 17 Oct 2002 17:16:42 +0200 (CEST) Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: Hello, On Thu, 17 Oct 2002, Gerald S. Williams wrote: > Here's a version of my previous algorithm with the checks added > back in to improve overall performance. This one runs about as > fast as the divide-and-compare algorithm for my test cases. :-) We might be running into code bloats thought. If we go for such big macros, I believe we should design them so that they run fast in the common case and call a helper function otherwise. Armin From guido@python.org Thu Oct 17 16:18:22 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 17 Oct 2002 11:18:22 -0400 Subject: [Python-Dev] PyEval_EvalCode() In-Reply-To: Your message of "Thu, 17 Oct 2002 10:38:39 +0200." <3DAE770F.5030304@isg.de> References: <3DAE770F.5030304@isg.de> Message-ID: <200210171518.g9HFIMc12781@odiug.zope.com> > This prototype is not available when including "Python.h" (Using v2.1.3) > Why did you exclude it from the "official" API prototypes? Because it's unlikely that someone writing a Python extension would ever have a need for it. > I found it in "eval.h", which has to be included manually, besides > "Python.h". > > Isn't that PyEval_EvalCode() the regular API function to run > CodeObjects from Py_CompileString() ? Probably -- I've never used it this way. But why aren't you using PyRun_String()? --Guido van Rossum (home page: http://www.python.org/~guido/) From gsw@agere.com Thu Oct 17 17:06:28 2002 From: gsw@agere.com (Gerald S. Williams) Date: Thu, 17 Oct 2002 12:06:28 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: Armin Rigo wrote: > We might be running into code bloats thought. If we go for such big > macros, I believe we should design them so that they run fast in the > common case and call a helper function otherwise. Easily done, but first you have to ask yourself if it's really ever more efficient than this: #define SAFE_MULTIPLY(dest,src1,src2,on_overflow)\ {\ size_t _x = src1;\ size_t _y = src2;\ size_t _dest = _x * _y;\ \ dest = _dest;\ \ if ((_x | _y) & HI_MASK)\ {\ if ((_dest / _x) != _y)\ {\ on_overflow;\ }\ }\ } If so, here's a macro/helper version: #define FULL_BITS (sizeof(size_t) * 8U) #define TOP_BIT (((size_t)1) << ((FULL_BITS)-1)) #define HALF_BITS (sizeof(size_t) * 4U) #define MID_BIT (((size_t)1) << ((HALF_BITS)-1)) #define LO_MASK ((((size_t)1) << (HALF_BITS))-1) #define HI_MASK (~(LO_MASK)) #define SAFE_MULTIPLY(dest,src1,src2,on_overflow)\ {\ size_t _x = src1;\ size_t _y = src2;\ size_t _dest = _x * _y;\ \ dest = _dest;\ \ if ((_x | _y) & HI_MASK)\ {\ if (_safe_multiply_check_for_overflow(_x,_y,_dest))\ {\ on_overflow;\ }\ }\ } int _safe_multiply_check_for_overflow( size_t x, size_t y, size_t dest) { size_t h; size_t l; if (x >= y) { h = x; l = y; } else { h = y; l = x; } if ((h & HI_MASK) && (l)) { int hlog; int llog; if (l & HI_MASK) { hlog = llog = FULL_BITS; } else { size_t mask; for ((mask=TOP_BIT),(hlog=FULL_BITS-1);!(mask&h);mask>>=1) { --hlog; } for ((mask=MID_BIT),(llog=HALF_BITS-1);!(mask&l);mask>>=1) { --llog; } } if ((hlog + llog > FULL_BITS - 1) || ((hlog + llog == FULL_BITS - 1) && !(dest & TOP_BIT))) { return 1; } } return 0; } Of course there are also the alternatives of using floating point numbers (assuming the mantissa has as many bits as size_t), or a double-width intermediate representation, if available. There are also some other tricks that can be used to optimize this solution. The time-consuming part is finding out if the sum of the log2's of the multiplicands (i.e., highest bit positions) is less than, greater than, or equal to - 1. But I said I was done tweaking for now. :-) -Jerry From Tino.Lange@isg.de Thu Oct 17 18:09:08 2002 From: Tino.Lange@isg.de (Tino Lange) Date: Thu, 17 Oct 2002 19:09:08 +0200 Subject: [Python-Dev] PyEval_EvalCode() References: <3DAE770F.5030304@isg.de> <200210171518.g9HFIMc12781@odiug.zope.com> Message-ID: <3DAEEEB4.4EFE10CB@isg.de> Guido van Rossum wrote: Hi! Thanks for your reply! > > PyEval_EvalCode() > > This prototype is not available when including "Python.h" (Using v2.1.3) > > Why did you exclude it from the "official" API prototypes? > Because it's unlikely that someone writing a Python extension would > ever have a need for it. Sure? I'd like to use it in a program that embeds a Python interpreter. The code snippets get compiled and checked by that function and can be called as a code object afterwards. > > Isn't that PyEval_EvalCode() the regular API function to run > > CodeObjects from Py_CompileString() ? > Probably -- I've never used it this way. At least a solution like that is described in the lastest "Programming Python" bible by Mark Lutz. (covering Python 2) > But why aren't you using > PyRun_String()? Speed, efficiency. If the code get's executed quite often (i. e. multiple times a second) it's worth precompiling (and maybe optimizing by Py_OptimizeFlag = 1), or? PyRun_String() will compile and check _every_ time - wasting a lot of resources. Cheers Tino From guido@python.org Thu Oct 17 18:29:00 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 17 Oct 2002 13:29:00 -0400 Subject: [Python-Dev] PyEval_EvalCode() In-Reply-To: Your message of "Thu, 17 Oct 2002 19:09:08 +0200." <3DAEEEB4.4EFE10CB@isg.de> References: <3DAE770F.5030304@isg.de> <200210171518.g9HFIMc12781@odiug.zope.com> <3DAEEEB4.4EFE10CB@isg.de> Message-ID: <200210171729.g9HHT0r21574@odiug.zope.com> > > > PyEval_EvalCode() > > > This prototype is not available when including "Python.h" (Using v2.1.3) > > > Why did you exclude it from the "official" API prototypes? > > Because it's unlikely that someone writing a Python extension would > > ever have a need for it. > > Sure? > I'd like to use it in a program that embeds a Python interpreter. > The code snippets get compiled and checked by that function and can be > called as a code object afterwards. > > > > Isn't that PyEval_EvalCode() the regular API function to run > > > CodeObjects from Py_CompileString() ? > > Probably -- I've never used it this way. > > At least a solution like that is described in the lastest "Programming > Python" bible by Mark Lutz. > (covering Python 2) > > > But why aren't you using > > PyRun_String()? > > Speed, efficiency. > If the code get's executed quite often (i. e. multiple times a second) > it's worth precompiling (and maybe optimizing by Py_OptimizeFlag = 1), > or? PyRun_String() will compile and check _every_ time - wasting a lot > of resources. Then feel free to import eval.h. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Thu Oct 17 19:15:27 2002 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 17 Oct 2002 14:15:27 -0400 Subject: [Python-Dev] creating struct-seq types from Python Message-ID: <15790.65087.725796.331958@grendel.zope.com> Martin, I've addressed this to you since you seem to be playing with the relevant code these days, but others may be interested as well. Have you looked at what it would take to create new "structure sequence" types from Python code? This seems like a reasonable thing to do, especially when an existing API returns a long(ish) tuple of fields. I'd like to be able to use it for my proposed urlparse changes (http://www.python.org/sf/624325), but there seems to be no way to get to it. What's needed to do something currently feels like too much code for what little is being done (see the patch if unsure of what I mean). Thanks! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tismer@tismer.com Thu Oct 17 19:24:56 2002 From: tismer@tismer.com (Christian Tismer) Date: Thu, 17 Oct 2002 11:24:56 -0700 Subject: [Python-Dev] Ann: Stackless technology merge under the way References: Message-ID: <3DAF0078.4060404@tismer.com> Armin Rigo wrote: > Hello Christian, > > On Wed, 16 Oct 2002, Christian Tismer wrote: > >>Stackless is changing heavily, once again. > > > Good news. It might be worth pointing out that even if continuations are > not directly supported, it should be possible to do them with thread > pickling. Unpickling a given thread state several times does the trick. This is correct. Actually, re-enabling continuations would not be *that* problem. What it needs it a copy-on-write like wrapper around frames, to make them duplicate when necessary. But I really don't want to go this path again, last but not least to avoid giving Guido an argument :-) a man cannot step into the same river twice -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From martin@v.loewis.de Thu Oct 17 19:28:54 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 17 Oct 2002 20:28:54 +0200 Subject: [Python-Dev] Re: creating struct-seq types from Python In-Reply-To: <15790.65087.725796.331958@grendel.zope.com> References: <15790.65087.725796.331958@grendel.zope.com> Message-ID: "Fred L. Drake, Jr." writes: > Have you looked at what it would take to create new "structure > sequence" types from Python code? Not really, but I think all you need to do is to expose PyStructSequence_InitType. I would recommend an interface like struct_seq(name, doc, n_in_sequence, (fields)) where fields is a list of (name,doc) tuples. You will need to temporarily allocate a PyStructSequence_Field array of len(fields)+1 elements, and put the PyStructSequence_Desc on the stack. You will also need to dynamically allocate a PyTypeObject which you return. I would put this into the new module. Regards, Martin From fdrake@acm.org Thu Oct 17 19:54:06 2002 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 17 Oct 2002 14:54:06 -0400 Subject: [Python-Dev] Re: creating struct-seq types from Python In-Reply-To: References: <15790.65087.725796.331958@grendel.zope.com> Message-ID: <15791.1870.843775.819662@grendel.zope.com> Martin v. Loewis writes: > Not really, but I think all you need to do is to expose > PyStructSequence_InitType. I would recommend an interface like > > struct_seq(name, doc, n_in_sequence, (fields)) Sounds like a plan! I've assigned this to me, since I'm the one asking for this. ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From skip@pobox.com Thu Oct 17 19:42:48 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 17 Oct 2002 13:42:48 -0500 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <15788.51819.500992.678200@gargle.gargle.HOWL> Message-ID: <15791.1192.105898.349109@montanaro.dyndns.org> BAW> Provide different field names for float values, e.g. f_mtime, BAW> f_ctime, f_atime, or maybe just mtime, atime, and ctime... ? Martin> The previous suggestion was st_mtimef, st_ctimef, ... Why not simply provide an alternative method, os.fstat()? Seems cleaner to me. The field names are hard enough to remember as it is... Skip From skip@pobox.com Thu Oct 17 19:56:47 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 17 Oct 2002 13:56:47 -0500 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: <15791.1192.105898.349109@montanaro.dyndns.org> References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <15788.51819.500992.678200@gargle.gargle.HOWL> <15791.1192.105898.349109@montanaro.dyndns.org> Message-ID: <15791.2031.219470.410776@montanaro.dyndns.org> BAW> Provide different field names for float values, e.g. f_mtime, BAW> f_ctime, f_atime, or maybe just mtime, atime, and ctime... ? Martin> The previous suggestion was st_mtimef, st_ctimef, ... Skip> Why not simply provide an alternative method, os.fstat()? Seems Skip> cleaner to me. The field names are hard enough to remember as it Skip> is... Doh! Engage brain before hitting the key... Of course there is already an fstat() function. I'll shut up now. Skip From gsw@agere.com Thu Oct 17 23:03:39 2002 From: gsw@agere.com (Gerald S. Williams) Date: Thu, 17 Oct 2002 18:03:39 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: I wrote: > But I said I was done tweaking for now. :-) Well, I was done tweaking, but then I decided to try out one more performance tweak. This is in desperate need of commenting, but it's finally faster than checking via division on my machine: #define FULL_BITS (sizeof(size_t) * 8U) #define TOP_BIT (((size_t)1) << ((FULL_BITS)-1)) #define HALF_BITS (sizeof(size_t) * 4U) #define MID_BIT (((size_t)1) << ((HALF_BITS)-1)) #define LO_MASK ((((size_t)1) << (HALF_BITS))-1) #define HI_MASK (~(LO_MASK)) #define Q_BITS (sizeof(size_t) * 2U) #define TQ_BITS (sizeof(size_t) * 6U) #define TQ_BIT (((size_t)1) << ((TQ_BITS)-1)) #define LTQ_MASK ((((size_t)1) << (TQ_BITS))-1) #define TQ_MASK (~(LTQ_MASK)) #define SAFE_MULTIPLY(dest,src1,src2,on_overflow)\ {\ size_t _x = src1;\ size_t _y = src2;\ size_t _dest = _x * _y;\ \ dest = _dest;\ \ if ((_x | _y) & HI_MASK)\ {\ if (_safe_multiply_check_for_overflow(_x,_y,_dest))\ {\ on_overflow;\ }\ }\ } int _safe_multiply_check_for_overflow( size_t h, size_t l, size_t dest) { if (l > h) { size_t temp = l; l = h; h = temp; } if ((h & HI_MASK) && (l)) { size_t lgt; size_t leq; if (l & HI_MASK) { lgt = (size_t)(-1); leq = 0; } else { size_t hbit; size_t mask; if (h & TQ_MASK) { for ((mask=TOP_BIT),(hbit=0);!(mask&h);mask>>=1) { ++hbit; } } else { for ((mask=TQ_BIT),(hbit=Q_BITS);!(mask&h);mask>>=1) { ++hbit; } } leq = 1 << hbit; mask = leq - 1; lgt = ~mask ^ leq; } if ((l & lgt) || ((l & leq) && !(dest & TOP_BIT))) { return 1; } } return 0; } -Jerry From tim.one@comcast.net Thu Oct 17 23:29:39 2002 From: tim.one@comcast.net (Tim Peters) Date: Thu, 17 Oct 2002 18:29:39 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: [Gerald S. Williams] > Well, I was done tweaking, but then I decided to try out one more > performance tweak. This is in desperate need of commenting, but > it's finally faster than checking via division on my machine: > ... If you've actually got the energy to time these things , how about adapting the int-mul overflow checking code in Python's intobject.c's int_mul() function too? The CVS (ditto 2.2.2) version + does rely on floating point + doesn't use division + does work for signed integral types + doesn't assume that the number of mantissa bits in a double is enough to hold a full-precision integral product + does assume that the largest integral product is within the dynamic range of a double It could be simplified a little if it were specialized to unsigned mult. From neal@metaslash.com Fri Oct 18 00:01:08 2002 From: neal@metaslash.com (Neal Norwitz) Date: Thu, 17 Oct 2002 19:01:08 -0400 Subject: [Python-Dev] Valgrind on 2.2.2 Message-ID: <20021017230108.GL9241@epoch.metaslash.com> I ran valgrind 1.0.3 on Python 2.2.2. Well, almost 2.2.2, it was from 12 Oct. After 90+ minutes and over 512 MB of RAM, there are no new/major issues to report. The complete valgrind log (231k) can be found here: http://www.metaslash.com/py/valgrind-2_2_2.txt I've cleaned up the log (200k) to remove some of the uninteresting stuff: http://www.metaslash.com/py/valgrind-clean-2_2_2.txt There are a few small memory leaks. I think most of the leaks are related to threads. The report contains memory still in use as well as leaks. To find memory leaks, search for ' lost ' without quotes. It's quite possible some of the memory still in use is due to reference leaks. Note: one common leak is from readline. Some of the read errors are from GLIBC. I had to skip test_pty, I think it caused the tests to hang. test_commands and test_popen2 fail when running valgrind. test_popen2 is due to valgrind output going to stdout/stderr. Neal From guido@python.org Fri Oct 18 01:38:40 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 17 Oct 2002 20:38:40 -0400 Subject: [Python-Dev] Valgrind on 2.2.2 In-Reply-To: Your message of "Thu, 17 Oct 2002 19:01:08 EDT." <20021017230108.GL9241@epoch.metaslash.com> References: <20021017230108.GL9241@epoch.metaslash.com> Message-ID: <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> > I ran valgrind 1.0.3 on Python 2.2.2. Well, almost 2.2.2, it was from > 12 Oct. After 90+ minutes and over 512 MB of RAM, there are no > new/major issues to report. > > The complete valgrind log (231k) can be found here: > http://www.metaslash.com/py/valgrind-2_2_2.txt > > I've cleaned up the log (200k) to remove some of the uninteresting stuff: > http://www.metaslash.com/py/valgrind-clean-2_2_2.txt I looked at this one only. > There are a few small memory leaks. I think most of the leaks are > related to threads. The report contains memory still in use as well > as leaks. To find memory leaks, search for ' lost ' without quotes. > It's quite possible some of the memory still in use is due to > reference leaks. This seems to be the most worrysome: ==28827== 268980 bytes in 15 blocks are possibly lost in loss record 100 of 106 ==28827== at 0x400487CD: realloc (vg_clientfuncs.c:270) ==28827== by 0x80994DC: _PyObject_GC_Resize (Modules/gcmodule.c:917) ==28827== by 0x80BD46E: PyFrame_New (Objects/frameobject.c:264) ==28827== by 0x80782C3: PyEval_EvalCodeEx (Python/ceval.c:2390) ==28827== by 0x807AA43: fast_function (Python/ceval.c:3173) ==28827== by 0x80779B5: eval_frame (Python/ceval.c:2034) ==28827== by 0x807892B: PyEval_EvalCodeEx (Python/ceval.c:2595) ==28827== by 0x807AA43: fast_function (Python/ceval.c:3173) ==28827== by 0x80779B5: eval_frame (Python/ceval.c:2034) ==28827== by 0x807892B: PyEval_EvalCodeEx (Python/ceval.c:2595) There are a few other records mentioning GC_Resize, but this one is the biggest. Could it be that the free frame list is botched? OTOH, what does "possibly lost" really mean? There are also a few fingers pointing in the direction of weakref_ref, e.g. ==28827== 520 bytes in 14 blocks are possibly lost in loss record 48 of 106 ==28827== at 0x400481B4: malloc (vg_clientfuncs.c:100) ==28827== by 0x8099519: _PyObject_GC_New (Modules/gcmodule.c:868) ==28827== by 0x8067BA5: PyWeakref_NewRef (Objects/weakrefobject.c:37) ==28827== by 0x8066119: add_subclass (Objects/typeobject.c:2249) ==28827== by 0x8061F29: PyType_Ready (Objects/typeobject.c:2219) ==28827== by 0x80605A8: type_new (Objects/typeobject.c:1280) ==28827== by 0x805EDA4: type_call (Objects/typeobject.c:183) ==28827== by 0x80ABB0C: PyObject_Call (Objects/abstract.c:1688) ==28827== by 0x807A34F: PyEval_CallObjectWithKeywords (Python/ceval.c:3058) ==28827== by 0x80AB0C6: PyObject_CallFunction (Objects/abstract.c:1679) Of course many of these could be caused by a single leak that drops a pointer to a container -- then everything owned by that container is also leaked. I noticed this one: ==28713== 572 bytes in 15 blocks are possibly lost in loss record 39 of 78 ==28713== at 0x400481B4: malloc (vg_clientfuncs.c:100) ==28713== by 0x8099519: _PyObject_GC_New (Modules/gcmodule.c:868) ==28713== by 0x80B2D09: PyMethod_New (Objects/classobject.c:2008) ==28713== by 0x80AF837: instance_getattr2 (Objects/classobject.c:702) ==28713== by 0x80AF73A: instance_getattr1 (Objects/classobject.c:676) ==28713== by 0x80B30F1: instance_getattr (Objects/classobject.c:715) ==28713== by 0x80577A2: PyObject_GetAttr (Objects/object.c:1108) ==28713== by 0x80B1731: half_cmp (Objects/classobject.c:1503) ==28713== by 0x80B1937: instance_compare (Objects/classobject.c:1572) ==28713== by 0x8055A6E: try_3way_compare (Objects/object.c:477) which led me to an easy-to-fix leak in half_cmp(), both in 2.2.2 and 2.3: diff -c -c -r2.154.8.1 classobject.c *** classobject.c 13 Jun 2002 21:36:35 -0000 2.154.8.1 --- classobject.c 18 Oct 2002 00:36:06 -0000 *************** *** 1507,1514 **** } args = Py_BuildValue("(O)", w); ! if (args == NULL) return -2; result = PyEval_CallObject(cmp_func, args); Py_DECREF(args); --- 1507,1516 ---- } args = Py_BuildValue("(O)", w); ! if (args == NULL) { ! Py_DECREF(cmp_func); return -2; + } result = PyEval_CallObject(cmp_func, args); Py_DECREF(args); but somehow I don't think that caused the report, because this exit can only be taken if there's a memory error. (Hm... or if w == NULL upon entry? How could that happen?) A similar one is on half_binop. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 18 01:51:17 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 17 Oct 2002 20:51:17 -0400 Subject: [Python-Dev] Valgrind on 2.2.2 In-Reply-To: Your message of "Thu, 17 Oct 2002 20:38:40 EDT." <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> References: <20021017230108.GL9241@epoch.metaslash.com> <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210180051.g9I0pH612923@pcp02138704pcs.reston01.va.comcast.net> > There are also a few fingers pointing in the direction of weakref_ref, > e.g. > > ==28827== 520 bytes in 14 blocks are possibly lost in loss record 48 of 106 > ==28827== at 0x400481B4: malloc (vg_clientfuncs.c:100) > ==28827== by 0x8099519: _PyObject_GC_New (Modules/gcmodule.c:868) > ==28827== by 0x8067BA5: PyWeakref_NewRef (Objects/weakrefobject.c:37) > ==28827== by 0x8066119: add_subclass (Objects/typeobject.c:2249) > ==28827== by 0x8061F29: PyType_Ready (Objects/typeobject.c:2219) > ==28827== by 0x80605A8: type_new (Objects/typeobject.c:1280) > ==28827== by 0x805EDA4: type_call (Objects/typeobject.c:183) > ==28827== by 0x80ABB0C: PyObject_Call (Objects/abstract.c:1688) > ==28827== by 0x807A34F: PyEval_CallObjectWithKeywords (Python/ceval.c:3058) > ==28827== by 0x80AB0C6: PyObject_CallFunction (Objects/abstract.c:1679) > > Of course many of these could be caused by a single leak that drops a > pointer to a container -- then everything owned by that container is > also leaked. This one seems simple: diff -c -c -r2.126.4.25 typeobject.c *** typeobject.c 11 Oct 2002 00:22:22 -0000 2.126.4.25 --- typeobject.c 18 Oct 2002 00:50:31 -0000 *************** *** 2249,2256 **** while (--i >= 0) { ref = PyList_GET_ITEM(list, i); assert(PyWeakref_CheckRef(ref)); ! if (PyWeakref_GET_OBJECT(ref) == Py_None) ! return PyList_SetItem(list, i, new); } i = PyList_Append(list, new); Py_DECREF(new); --- 2249,2259 ---- while (--i >= 0) { ref = PyList_GET_ITEM(list, i); assert(PyWeakref_CheckRef(ref)); ! if (PyWeakref_GET_OBJECT(ref) == Py_None) { ! i = PyList_SetItem(list, i, new); ! Py_DECREF(new); ! return i; ! } } i = PyList_Append(list, new); Py_DECREF(new); --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Fri Oct 18 09:25:13 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: Fri, 18 Oct 2002 10:25:13 +0200 Subject: [Python-Dev] Incorporating bsddb3 Message-ID: <200210180825.g9I8PDK7015010@mira.informatik.hu-berlin.de> I have changes in my sandbox to incorporate bsddb3 into Python proper. This essentially is a copy of bsddb3 3.4.0, with the following modifications: - the extension module is called _bsddb (not bsddb3._db) - the package is called bsddb (not bsddb3) Both test_bsddb and test_anydbm continue to work unmodified. Is it ok to commit these changes? Regards, Martin From martin@v.loewis.de Fri Oct 18 07:37:45 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 18 Oct 2002 08:37:45 +0200 Subject: [Python-Dev] float atime/mtime/ctime - a bad idea? In-Reply-To: <15791.1192.105898.349109@montanaro.dyndns.org> References: <200210160047.g9G0lHx20255@pcp02138704pcs.reston01.va.comcast.net> <15788.51819.500992.678200@gargle.gargle.HOWL> <15791.1192.105898.349109@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > Martin> The previous suggestion was st_mtimef, st_ctimef, ... > > Why not simply provide an alternative method, os.fstat()? Because os.fstat already exists, and also has the option of either returning floating point time stamps or integer time stamps - just like os.stat and os.lstat. Regards, Martin From guido@python.org Fri Oct 18 15:03:40 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 18 Oct 2002 10:03:40 -0400 Subject: [Python-Dev] Valgrind on 2.2.2 In-Reply-To: Your message of "Thu, 17 Oct 2002 20:51:17 EDT." <200210180051.g9I0pH612923@pcp02138704pcs.reston01.va.comcast.net> References: <20021017230108.GL9241@epoch.metaslash.com> <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> <200210180051.g9I0pH612923@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210181403.g9IE3fO17121@pcp02138704pcs.reston01.va.comcast.net> > This one seems simple: [patch to typeobject.c] Deceptively so -- I forgot that PyList_SetItem(list, i, new) steals a reference to new. :-( The search is still on... --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Fri Oct 18 23:02:44 2002 From: skip@pobox.com (Skip Montanaro) Date: Fri, 18 Oct 2002 17:02:44 -0500 Subject: [Python-Dev] To -I(nclude) or not to -I(nclude), that is the question... Message-ID: <15792.34052.962582.864797@montanaro.dyndns.org> In installing mxBase 2.0.4 on my MacOS 10.2.1 system I get warnings like the following: gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -Wno-long-double -no-cpp-precomp -Imx/Queue/mxQueue -I/Users/skip/local/include/python2.3 -I/usr/include -I/usr/local/include -c mx/Queue/mxQueue/mxQueue.c -o build/temp.darwin-6.1-Power Macintosh-2.3/mx/Queue/mxQueue/mxQueue/mxQueue.o cc1: warning: changing search order for system directory "/usr/local/include" cc1: warning: as it has already been specified as a non-system directory cc1: warning: changing search order for system directory "/usr/include" cc1: warning: as it has already been specified as a non-system directory This warning bothers me a bit, as it suggests I'm screwing up the compiler's notions about header file search order. Has anyone else seen this and investigated how to get rid of this problem? This is related to bug http://python.org/sf/589427 (which was assigned to me). It's due to the gen_preprocess_options() function in distutils/ccompiler.py. This warning seems to be related to gcc version >= 3.1. I have a quick hack in my local copy of distutils/ccompiler.py. At the bottom of gen_preprocess_options() I replaced for dir in include_dirs: pp_opts.append ("-I%s" % dir) with pp_opts.extend (gen_preprocess_includes(include_dirs)) and added these two functions to the file: def gen_preprocess_includes_macosx_gcc(dirs): """GCC on MacOSX complains if /usr/include or /usr/local/include are mentioned in -I. """ pp_opts = [] for dir in dirs: if dir not in ("/usr/include", "/usr/local/include"): pp_opts.append ("-I%s" % dir) return pp_opts def gen_preprocess_includes(dirs): """Generate the -I flags for a compile command.""" if sys.platform == "darwin": return gen_preprocess_includes_macosx_gcc(dirs) pp_opts = [] for dir in dirs: pp_opts.append ("-I%s" % dir) return pp_opts This is an obscure solution, at best. I'd prefer to at least test the compiler and version. How can I tell what compiler and version will be used to compile files, or can't I can this level? (Seems to me that gen_preprocess_options begs to be a method of the CCompiler class.) Thx, Skip From barry@python.org Fri Oct 18 23:10:55 2002 From: barry@python.org (Barry A. Warsaw) Date: Fri, 18 Oct 2002 18:10:55 -0400 Subject: [Python-Dev] To -I(nclude) or not to -I(nclude), that is the question... References: <15792.34052.962582.864797@montanaro.dyndns.org> Message-ID: <15792.34543.263408.609946@gargle.gargle.HOWL> >>>>> "SM" == Skip Montanaro writes: SM> This warning bothers me a bit, as it suggests I'm screwing up SM> the compiler's notions about header file search order. Has SM> anyone else seen this and investigated how to get rid of this SM> problem? This is related to bug http://python.org/sf/589427 SM> (which was assigned to me). It's due to the SM> gen_preprocess_options() function in distutils/ccompiler.py. SM> This warning seems to be related to gcc version >= 3.1. I seem to remember that Martin said the gcc folks agreed that this behavior was a bug on their part. I tried building Python on Linux with gcc 3.1 several months ago and had the same problems. I dropped back to 2.96 and haven't looked back. :) Search around in python-dev for a discussion on this, but unfortunately I don't remember the exact time frame. -Barry From brian@sweetapp.com Sat Oct 19 01:10:55 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Fri, 18 Oct 2002 17:10:55 -0700 Subject: [Python-Dev] Weakref design questions In-Reply-To: <15792.34543.263408.609946@gargle.gargle.HOWL> Message-ID: <000401c27703$fe5cd860$21795418@dell1700> 1. Is there any reason why builtin methods cannot be proxied? 2. It would be handy for my application if a callback could be triggered when an object has no more weak references attached to it. It seems like my application could be a fairly common one: # C library pseudocode def c_library_func(): # C code while 1: o = create_complex_object() user_call_back(o) del o # Python bindings pseudocode def python_bindings_user_call_back (o): # C code py_o = create_python_wrapper_object(o) proxy = PyWeakref_NewProxy(py_o) python_function(py_o) Py_DECREF(proxy) Py_DECREF(py_o) # This will kill the proxy # Evil Python user code evil = None def python_function(o): global evil o.foo() evil = o start(python_function) evil.foo() # Nice exception because evil is a dead proxy # More evil Python user code more_evil = None def python_function(o): global more_evil o.foo() more_evil = o.foo start(python_function) more_evil() # Crash because the underlying data structures that # the Python wrapper object depends on are dead My current solution to this problem is to create my own callable object type that supports weakrefs. That object is then used to wrap the real bound method object e.g. def getattr(self, name): # This is C code callable = MyCallAble_New(Py_FindMethod(...); objects_to_kill_after_py_func_call.add(callable); return PyWeakref_NewProxy(callable); Avoiding this hassle is the reason for my question. Pruning callable objects that the user is done with is the reason for my request. Cheers, Brian From guido@python.org Sat Oct 19 01:23:50 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 18 Oct 2002 20:23:50 -0400 Subject: [Python-Dev] Weakref design questions In-Reply-To: Your message of "Fri, 18 Oct 2002 17:10:55 PDT." <000401c27703$fe5cd860$21795418@dell1700> References: <000401c27703$fe5cd860$21795418@dell1700> Message-ID: <200210190023.g9J0Nou25728@pcp02138704pcs.reston01.va.comcast.net> > 1. Is there any reason why builtin methods cannot be proxied? The instances of a type need to have a pointer added if the type is to support weak refs to its instances. We've only done this for the obvious candidates like user-defined class instances and a few others. You must weigh the cost of the extra pointer vs. the utility of being able to use weak refs for a particular type. > 2. It would be handy for my application if a callback could be triggered > when an object has no more weak references attached to it. > > It seems like my application could be a fairly common one: > > # C library pseudocode > def c_library_func(): # C code > while 1: > o = create_complex_object() > user_call_back(o) > del o > > # Python bindings pseudocode > def python_bindings_user_call_back (o): # C code > py_o = create_python_wrapper_object(o) > proxy = PyWeakref_NewProxy(py_o) > python_function(py_o) > Py_DECREF(proxy) > Py_DECREF(py_o) # This will kill the proxy > > > # Evil Python user code > evil = None > def python_function(o): > global evil > o.foo() > evil = o > > start(python_function) > evil.foo() # Nice exception because evil is a dead proxy > > # More evil Python user code > > more_evil = None > def python_function(o): > global more_evil > o.foo() > more_evil = o.foo > > start(python_function) > more_evil() # Crash because the underlying data structures that > # the Python wrapper object depends on are dead > > My current solution to this problem is to create my own callable object > type that supports weakrefs. That object is then used to wrap the real > bound method object e.g. > > def getattr(self, name): # This is C code > callable = MyCallAble_New(Py_FindMethod(...); > objects_to_kill_after_py_func_call.add(callable); > return PyWeakref_NewProxy(callable); > > Avoiding this hassle is the reason for my question. > > Pruning callable objects that the user is done with is the reason for my > request. I'm sorry, but I don't understand why you're using weak references here at all. Is it really the proxy function that you're after? If you're worried about the complex object 'o' being referenced after the C library has killed it (a ligitimate concern), while the Python wrapper can be kept alive using the scenario you show, the Python wrapper py_o should set its pointer to 'o' to NULL when o's lifetime ends (or a little before) and make sure that methods on the wrapper raise an exception when the reference is NULL. This is how all well-behaved wrapper objects behave; e.g. file and socket objects in Python check if the file is closed in each method implementation. --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Sat Oct 19 01:39:04 2002 From: dave@boost-consulting.com (David Abrahams) Date: 18 Oct 2002 20:39:04 -0400 Subject: [Python-Dev] Weakref design questions In-Reply-To: <000401c27703$fe5cd860$21795418@dell1700> References: <000401c27703$fe5cd860$21795418@dell1700> Message-ID: Brian Quinlan writes: > 1. Is there any reason why builtin methods cannot be proxied? > > 2. It would be handy for my application if a callback could be triggered > when an object has no more weak references attached to it. > > It seems like my application could be a fairly common one: > > # C library pseudocode > def c_library_func(): # C code > while 1: > o = create_complex_object() > user_call_back(o) > del o > > # Python bindings pseudocode > def python_bindings_user_call_back (o): # C code > py_o = create_python_wrapper_object(o) > proxy = PyWeakref_NewProxy(py_o) > python_function(py_o) > Py_DECREF(proxy) > Py_DECREF(py_o) # This will kill the proxy What's the point of the proxy? You never use it for anything. I guess you must've meant: python_function(proxy) above. > # Evil Python user code > evil = None > def python_function(o): > global evil > o.foo() > evil = o > > start(python_function) > evil.foo() # Nice exception because evil is a dead proxy Hum. In Boost.Python the default is to copy the C++ object when passing it to a Python callback, though it's possible for the user to explicitly say "just build a Python object around a reference to the C++ object -- Python code beware of lifetime issues". I guess I like your idea, though, as a safer alternative for non-copyable or for very expensive-to-copy C++ objects. > # More evil Python user code > > more_evil = None > def python_function(o): > global more_evil > o.foo() > more_evil = o.foo > > start(python_function) > more_evil() # Crash because the underlying data structures that > # the Python wrapper object depends on are dead Hum. >>> class userlist(list): pass # just to get weakref ability ... >>> l = userlist() + [ 1, 2, 3 ] >>> type(l) # Interesting (buggy?) behavior with 2.2.1 >>> l = userlist() # Try again >>> l += [1,2,3] >>> type(l) # Better >>> p = weakref.proxy(l) >>> a = p.append >>> p [1, 2, 3] >>> a >>> del l >>> p # The list is still alive! [1, 2, 3] >>> a # Because it (not the proxy) is bound into a >>> a(2) # a works without crashing >>> p [1, 2, 3, 2] How does this differ from what's happening for you? > Pruning callable objects that the user is done with is the reason > for my request. Hmm, it kinda seems like you want to prune some of them before the user's done with them. Don't users expect objects to stay alive when they hold references? -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Building C/C++ Extensions for Python: Dec 9-11, Austin, TX http://www.enthought.com/training/building_extensions.html From brian@sweetapp.com Sat Oct 19 02:00:28 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Fri, 18 Oct 2002 18:00:28 -0700 Subject: [Python-Dev] Weakref design questions In-Reply-To: <200210190023.g9J0Nou25728@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <000501c2770a$ea8fb4e0$21795418@dell1700> Guido van Rossum wrote: > The instances of a type need to have a pointer added if the > type is to support weak refs to its instances. We've only > done this for the obvious candidates like user-defined class > instances and a few others. You must weigh the cost of the > extra pointer vs. the utility of being able to use weak refs > for a particular type. Fair enough. > I'm sorry, but I don't understand why you're using weak references > here at all. Is it really the proxy function that you're after? I don't understand the question. > If > you're worried about the complex object 'o' being referenced after the > C library has killed it (a ligitimate concern), while the Python > wrapper can be kept alive using the scenario you show, the Python > wrapper py_o should set its pointer to 'o' to NULL when o's lifetime > ends (or a little before) and make sure that methods on the wrapper > raise an exception when the reference is NULL. That is definitely one possible way to do it. However, I am wrapping a complete DOM, with dozens of objects containing, collectively, hundreds of methods. Adding an explicit check to each method seemed like a lot more pain than using proxies. Avoiding explicit checks also offers a potential performance advantage because sometimes the objects are owned and no checking is required. In that case I simply return the object directly without using a proxy. > This is how all well-behaved wrapper objects behave; e.g. file and > socket objects in Python check if the file is closed in each method > implementation. They contain far fewer methods. Also, there is more centralization is file and socket objects i.e. they know themselves if they are in a valid state or not, this is not true of my DOM objects. Cheers, Brian From brian@sweetapp.com Sat Oct 19 02:21:23 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Fri, 18 Oct 2002 18:21:23 -0700 Subject: [Python-Dev] Weakref design questions In-Reply-To: Message-ID: <000601c2770d$d6c80c70$21795418@dell1700> David Abrahams: > What's the point of the proxy? You never use it for anything. I guess > you must've meant: > python_function(proxy) > above. Nice catch. > Hum. In Boost.Python the default is to copy the C++ object when > passing it to a Python callback, though it's possible for the user > to explicitly say "just build a Python object around a reference to > the C++ object -- Python code beware of lifetime issues". I guess I > like your idea, though, as a safer alternative for non-copyable or > for very expensive-to-copy C++ objects. DOMs, the data structures in question, can be obscenely expensive to copy. And I firmly believe that Python code should never be able to cause a crash. [nice demonstration deleted] > How does this differ from what's happening for you? When you created the bound method p.append, you caused the reference count of p to be increased. In my case, since Python is not capable of altering the lifetime of p (it's dead with the C library says that it is dead), this is a bad thing. So I proxy the bound methods and kill them when p dies. > Hmm, it kinda seems like you want to prune some of them before the > user's done with them. I actually presented two different desires at once and know I'm paying the price. There are two reasons that I might want to delete a proxied object: 1. Its lifetime is over and I don't want anyone to access it 2. There are no proxy objects that refer to it, making it inaccessible (since I hold the only actual reference) Reason #1 is why I could like builtin methods to support weakrefs. Reason #2 is why I would like a callback when the last weakref dies. > Don't users expect objects to stay alive when they hold references? Probably. But that is not always an option. Cheers, Brian From dave@boost-consulting.com Sat Oct 19 02:42:28 2002 From: dave@boost-consulting.com (David Abrahams) Date: 18 Oct 2002 21:42:28 -0400 Subject: [Python-Dev] Weakref design questions In-Reply-To: <000601c2770d$d6c80c70$21795418@dell1700> References: <000601c2770d$d6c80c70$21795418@dell1700> Message-ID: Brian Quinlan writes: > David Abrahams: > > What's the point of the proxy? You never use it for anything. I guess > > you must've meant: > > > python_function(proxy) > > > above. > > Nice catch. > > > Hum. In Boost.Python the default is to copy the C++ object when > > passing it to a Python callback, though it's possible for the user > > to explicitly say "just build a Python object around a reference to > > the C++ object -- Python code beware of lifetime issues". I guess I > > like your idea, though, as a safer alternative for non-copyable or > > for very expensive-to-copy C++ objects. > > DOMs, the data structures in question, can be obscenely expensive to > copy. This may not be possible for you, but in C++ That would generally be a "handle" class with reference semantics (using a reference-counted smart pointer, for example) so that "copying" the DOM really just amounted to bumping a reference count somewhere. > And I firmly believe that Python code should never be able to cause a > crash. Me too, but some of my users insist on being able to trade safety for performance. > [nice demonstration deleted] > > How does this differ from what's happening for you? > > When you created the bound method p.append, you caused the reference > count of p to be increased. I think I caused the reference count of l to be increased. AFAIK, p isn't keeping the list alive. > In my case, since Python is not capable of altering the lifetime of p > (it's dead with the C library says that it is dead), this is a bad > thing. Yeah, I see that now. > So I proxy the bound methods and kill them when p dies. I don't quite understand what "proxy the bound methods" means. > > Hmm, it kinda seems like you want to prune some of them before the > > user's done with them. > > I actually presented two different desires at once and know I'm paying > the price. > > There are two reasons that I might want to delete a proxied object: > > 1. Its lifetime is over and I don't want anyone to access it OK. > 2. There are no proxy objects that refer to it, making it inaccessible > (since I hold the only actual reference) > > Reason #1 is why I could like builtin methods to support weakrefs. > Reason #2 is why I would like a callback when the last weakref dies. So you want a weakrefref. Well, I think this problem can be solved by applying the Fundamental Theorem of Software Engineering: apply an extra level of indirection. But Guido's approach of having the wrapper methods check for a null DOM* seems like a reasonable one to me. Incidentally, that's what you automatically get from Boost.Python when your object is held by std::auto_ptr, which is a smart pointer class that can release ownership. I think I want to do something similar for boost::weak_ptr, but I haven't gotten around to it yet. > > Don't users expect objects to stay alive when they hold references? > > Probably. But that is not always an option. That's clear to me now. Very interesting thread; thanks for posting it here! -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Building C/C++ Extensions for Python: Dec 9-11, Austin, TX http://www.enthought.com/training/building_extensions.html From brian@sweetapp.com Sat Oct 19 03:51:18 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Fri, 18 Oct 2002 19:51:18 -0700 Subject: [Python-Dev] Weakref design questions In-Reply-To: Message-ID: <000c01c2771a$66023170$21795418@dell1700> David Abrahams: > > So I proxy the bound methods and kill them when p dies. > I don't quite understand what "proxy the bound methods" means. Most extension types have a function like this: PyObject * MyObject_getattr(MyObject* self, char * name) { /* check for attributes */ ... /* ok, not an attribute, now check the methods return Py_FindMethod( MyObject_methods, (PyObject *) self, name); } Py_FindMethod() returns a PyCFunctionObject. The PyCFunctionObject will own a reference to the MyObject "self". Since "self" has a limited lifetime this would be bad. So we could do this: PyObject * MyObject_getattr(MyObject* self, char * name) { /* check for attributes */ ... /* ok, not an attribute, now check the methods method = Py_FindMethod( MyObject_methods, (PyObject *) self, name); if (method != NULL) { add_to_list_of_objects_to_kill(method) return PyWeakref_NewProxy(method); } return NULL; } But we can't quite do this because builtin functions are not proxyable. > So you want a weakrefref. Well, I think this problem can be solved > by applying the Fundamental Theorem of Software Engineering: apply > an extra level of indirection. That's what I do. I was just wondering if there is any reason not to make C functions weakref/proxyable. > But Guido's approach of having the wrapper methods check for a null DOM* > seems like a reasonable one to me. It's not unreasonable and it is a bit simpler. But it is more work and reduces performance slightly in the case where no check need be performed i.e. the object is owned by Python. > That's clear to me now. Very interesting thread; thanks for posting > it here! Yeah, my first non-stupid post to python-dev. Cheers, Brian From martin@v.loewis.de Sat Oct 19 04:39:09 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 19 Oct 2002 05:39:09 +0200 Subject: [Python-Dev] Weakref design questions In-Reply-To: <000501c2770a$ea8fb4e0$21795418@dell1700> References: <000501c2770a$ea8fb4e0$21795418@dell1700> Message-ID: Brian Quinlan writes: > That is definitely one possible way to do it. However, I am wrapping a > complete DOM, with dozens of objects containing, collectively, hundreds > of methods. > > Adding an explicit check to each method seemed like a lot more pain than > using proxies. You don't have to add it to every method. You can perform the check in tp_getattro before performing the method lookup. Alternatively, you can change the ob_type of the object to simply drop the methods that are not available anymore. Regards, Martin From brian@sweetapp.com Sat Oct 19 04:54:45 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Fri, 18 Oct 2002 20:54:45 -0700 Subject: [Python-Dev] Weakref design questions In-Reply-To: Message-ID: <000d01c27723$432da220$21795418@dell1700> Martin wrote: > You don't have to add it to every method. You can perform the check in > tp_getattro before performing the method lookup. That would be dangerous! See my original "more evil" example. > Alternatively, you can change the ob_type of the object to simply drop > the methods that are not available anymore. I like this strategy! But I still think that this is more painful/less elegant than using proxies. Cheers, Brian From dave@boost-consulting.com Sat Oct 19 04:46:45 2002 From: dave@boost-consulting.com (David Abrahams) Date: 18 Oct 2002 23:46:45 -0400 Subject: [Python-Dev] Weakref design questions In-Reply-To: <000d01c27723$432da220$21795418@dell1700> References: <000d01c27723$432da220$21795418@dell1700> Message-ID: Brian Quinlan writes: > Martin wrote: > > You don't have to add it to every method. You can perform the check in > > tp_getattro before performing the method lookup. > > That would be dangerous! See my original "more evil" example. Yeah, doesn't work for that case. > > Alternatively, you can change the ob_type of the object to simply drop > > > the methods that are not available anymore. > > I like this strategy! But I still think that this is more painful/less > elegant than using proxies. I don't see how that works either. If you have two objects of the same type, they may die at different times. If the type drops its methods all the objects become disabled. Furthermore, I think it still doesn't help with "more evil", since nobody's touching the type at that point - the method has already been looked up and kept alive by binding it to the underlying object. -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Building C/C++ Extensions for Python: Dec 9-11, Austin, TX http://www.enthought.com/training/building_extensions.html From brian@sweetapp.com Sat Oct 19 05:03:10 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Fri, 18 Oct 2002 21:03:10 -0700 Subject: [Python-Dev] Weakref design questions In-Reply-To: <000d01c27723$432da220$21795418@dell1700> Message-ID: <000e01c27724$7098df30$21795418@dell1700> Brian Quinlan wrote: >> Alternatively, you can change the ob_type of the object to simply >> drop the methods that are not available anymore. > I like this strategy! But I still think that this is more > painful/less elegant than using proxies. I'm an idiot. This won't work either for the same reason that I gave before. Cheers, Brian From martin@v.loewis.de Sat Oct 19 05:18:12 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 19 Oct 2002 06:18:12 +0200 Subject: [Python-Dev] Weakref design questions In-Reply-To: <000401c27703$fe5cd860$21795418@dell1700> References: <000401c27703$fe5cd860$21795418@dell1700> Message-ID: Brian Quinlan writes: > My current solution to this problem is to create my own callable object > type that supports weakrefs. That object is then used to wrap the real > bound method object e.g. > > def getattr(self, name): # This is C code > callable = MyCallAble_New(Py_FindMethod(...); > objects_to_kill_after_py_func_call.add(callable); > return PyWeakref_NewProxy(callable); > > Avoiding this hassle is the reason for my question. I notice that you don't need the Weakref property of those proxies at all: You know precisely the set of all objects to consider when the underlying C object has gone away. So you merely want a proxy, not a weak proxy: both for the entire object, and for the methods. So for this code, you can save the proxy, and return your callable object. Make the null pointer check in its tp_call slot, and don't kill it after py func call, but merely clear the pointer. I also notice that you rely on the fact that Python code has no way to find out the underlying object of a weak proxy. I think this is a weak assumption - there is no guarantee that this is not possible, or might not be possible in the future. Regards, Martin From brian@sweetapp.com Sat Oct 19 05:44:39 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Fri, 18 Oct 2002 21:44:39 -0700 Subject: [Python-Dev] Weakref design questions In-Reply-To: Message-ID: <001101c2772a$3bef5e70$21795418@dell1700> Martin wrote: > So you merely want a proxy, not a weak proxy: both for the > entire object, and for the methods. So for this code, you > can save the proxy, and return your callable object. Make > the null pointer check in its tp_call slot, and don't kill > it after py func call, but merely clear the pointer. As you say, I need a proxy for my methods and a proxy for my objects. Creating my own proxy type for methods is easy, since the only interesting thing that must be proxied is __call__. Creating my own proxy type for arbitrary objects is harder because I must create a proxy for every slot that I use. That requires duplication a lot of the work that already lives in weakrefobject.c Also, for both method and object proxies, I have to invent a mechanism to signal that the original object is dead. weakrefobject.c already defines a nice mechanism. > I also notice that you rely on the fact that Python code has no way > to find out the underlying object of a weak proxy. I think this is a weak > assumption - there is no guarantee that this is not possible, or might > not be possible in the future. This is a concern. If I am mistaken in my assumption, please let me know and I will either invent my own proxy type or use the check-on-every-method-call technique. Cheers, Brian From martin@v.loewis.de Sat Oct 19 07:37:04 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 19 Oct 2002 08:37:04 +0200 Subject: [Python-Dev] Weakref design questions In-Reply-To: <001101c2772a$3bef5e70$21795418@dell1700> References: <001101c2772a$3bef5e70$21795418@dell1700> Message-ID: Brian Quinlan writes: > > I also notice that you rely on the fact that Python code has no > > way to find out the underlying object of a weak proxy. I think > > this is a weak assumption - there is no guarantee that this is not > > possible, or might not be possible in the future. > This is a concern. If I am mistaken in my assumption, please let me know > and I will either invent my own proxy type or use the > check-on-every-method-call technique. I'm not exactly sure what your assumption is. If it is "in Python 2.3, there is no way to unwrap a proxy except by writing an extension module", then your assumption is correct. If your assumption is "there is a guarantee that there never will be such mechanism", your assumption is incorrect. Regards, Martin From mal@lemburg.com Sat Oct 19 11:12:34 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 19 Oct 2002 12:12:34 +0200 Subject: [Python-Dev] Re: Moving to Unicode 3.2 ([Python-checkins] python/dist/src/Objects unicodectype.c,2.11,2.12 unicodetype_db.h,1.4,1.5) References: Message-ID: <3DB13012.3040604@lemburg.com> loewis@users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/Objects > In directory usw-pr-cvs1:/tmp/cvs-serv10274/Objects > > Modified Files: > unicodectype.c unicodetype_db.h > Log Message: > Update to Unicode 3.2 database. I haven't seen any messages about this on python-dev. Did I miss something ? The switch from Unicode 3.0 is a big one since 3.2 introduces non-BMP character points for the first time. I also don't think that it is a good idea to ship the Unicode 3.2 database while the code behaves as defined in Unicode 3.0. And last not least, I'd like to be asked before you make such changes. Thanks, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mal@lemburg.com Sat Oct 19 11:18:43 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 19 Oct 2002 12:18:43 +0200 Subject: [Python-Dev] To -I(nclude) or not to -I(nclude), that is the question... References: <15792.34052.962582.864797@montanaro.dyndns.org> Message-ID: <3DB13183.9000206@lemburg.com> Skip Montanaro wrote: > In installing mxBase 2.0.4 on my MacOS 10.2.1 system I get warnings like the > following: > > gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -Wno-long-double -no-cpp-precomp -Imx/Queue/mxQueue -I/Users/skip/local/include/python2.3 -I/usr/include -I/usr/local/include -c mx/Queue/mxQueue/mxQueue.c -o build/temp.darwin-6.1-Power Macintosh-2.3/mx/Queue/mxQueue/mxQueue/mxQueue.o > cc1: warning: changing search order for system directory "/usr/local/include" > cc1: warning: as it has already been specified as a non-system directory > cc1: warning: changing search order for system directory "/usr/include" > cc1: warning: as it has already been specified as a non-system directory > > This warning bothers me a bit, as it suggests I'm screwing up the compiler's > notions about header file search order. Has anyone else seen this and > investigated how to get rid of this problem? I've had a few bug reports related to this, but all of them were from Solaris users. The -I/usr/include causes the GCC compiler to pick up a system stdarg.h header file which causes a compile error (GCC ships with its own stdarg.h files). The report for MacOS is new, though. Perhaps this is a generic GCC problem ? (#include should look in the compiler dirs first and only then scan the additional -I paths) > This is related to bug > http://python.org/sf/589427 (which was assigned to me). It's due to the > gen_preprocess_options() function in distutils/ccompiler.py. This warning > seems to be related to gcc version >= 3.1. > > I have a quick hack in my local copy of distutils/ccompiler.py. At the > bottom of gen_preprocess_options() I replaced > > for dir in include_dirs: > pp_opts.append ("-I%s" % dir) > > with > > pp_opts.extend (gen_preprocess_includes(include_dirs)) > > and added these two functions to the file: > > def gen_preprocess_includes_macosx_gcc(dirs): > """GCC on MacOSX complains if /usr/include or /usr/local/include are > mentioned in -I. > """ > pp_opts = [] > for dir in dirs: > if dir not in ("/usr/include", "/usr/local/include"): > pp_opts.append ("-I%s" % dir) > return pp_opts > > def gen_preprocess_includes(dirs): > """Generate the -I flags for a compile command.""" > if sys.platform == "darwin": > return gen_preprocess_includes_macosx_gcc(dirs) > > pp_opts = [] > for dir in dirs: > pp_opts.append ("-I%s" % dir) > return pp_opts > > This is an obscure solution, at best. I'd prefer to at least test the > compiler and version. How can I tell what compiler and version will be used > to compile files, or can't I can this level? (Seems to me that > gen_preprocess_options begs to be a method of the CCompiler class.) > > Thx, > > Skip > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Sat Oct 19 11:54:44 2002 From: guido@python.org (Guido van Rossum) Date: Sat, 19 Oct 2002 06:54:44 -0400 Subject: [Python-Dev] Weakref design questions In-Reply-To: Your message of "Sat, 19 Oct 2002 05:39:09 +0200." References: <000501c2770a$ea8fb4e0$21795418@dell1700> Message-ID: <200210191054.g9JAsiG26920@pcp02138704pcs.reston01.va.comcast.net> [Brian] > > That is definitely one possible way to do it. However, I am > > wrapping a complete DOM, with dozens of objects containing, > > collectively, hundreds of methods. > > > > Adding an explicit check to each method seemed like a lot more > > pain than using proxies. [Martin] > You don't have to add it to every method. You can perform the check in > tp_getattro before performing the method lookup. Alternatively, you > can change the ob_type of the object to simply drop the methods that > are not available anymore. No, his second example defeated that: def evil_python_callback(o): global evil_method evil_method = o.meth def called_later(): evil_method() # SegFault This does the getattr when it's allowed, but makes the call later. To Brian: I really don't want to make bound methods proxyable. (And it wouldn't help you anyway until Python 2.3 is released.) If you really don't want to modify each method, you can write your own callable type and use that to wrap the methods; this can then check whether the object is still alive before passing on the call. If you think this is too expensive, go back to checking in each method -- since a check is needed anyway, you can't get cheaper than that. (A weakref proxy isn't free either.) To signal invalidity, the Python wrapper would have to set the 'o' pointer to NULL. I also agree with Martin's observation that weakref proxies aren't intended to make the original object undiscoverable. --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Sat Oct 19 16:06:43 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 19 Oct 2002 17:06:43 +0200 Subject: [Python-Dev] Re: Moving to Unicode 3.2 In-Reply-To: <3DB13012.3040604@lemburg.com> References: <3DB13012.3040604@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > I haven't seen any messages about this on python-dev. Did I miss > something ? No. For a change like this, I did not think consultation was necessary. > The switch from Unicode 3.0 is a big one since 3.2 introduces > non-BMP character points for the first time. I disagree; it's a small change. Just look at the patch itself: apart from the (considerably large) generated data, there were very few actual changes to source code: Changing a few limits was sufficient. Since there are no backwards compatibility issues, and no design choices (apart from the choice of updating the database at all), this is a straight-forward change. > I also don't think that it is a good idea to ship the Unicode > 3.2 database while the code behaves as defined in Unicode 3.0. Can you please elaborate? What code behaves as defined in Unicode 3.0 that is incompatible with the Unicode 3.2 database? > And last not least, I'd like to be asked before you make such > changes. I find this quite a possessive view, and I would prefer if you bring up technical arguments instead of procedural ones, but ok... I was under the impression that I can apply my own professional judgement when deciding what patches to apply without consultation, in what cases to ask on python-dev, and when to submit a patch to SF. Apparently, this impression is wrong. Can you please give precise instructions what constitutes "such a change"? Also, should I back this change out? Regards, Martin From guido@python.org Sat Oct 19 16:42:12 2002 From: guido@python.org (Guido van Rossum) Date: Sat, 19 Oct 2002 11:42:12 -0400 Subject: [Python-Dev] Gone for a week Message-ID: <200210191542.g9JFgDp28012@pcp02138704pcs.reston01.va.comcast.net> I'm going to disappear for a week; back on the 28th. I'll be on a cruise ship teaching Python and Zope. (Can you say boondoggle? It's spelled www.geekcruises.com. :-) They've threatened to hook us up with internet access to maybe I won't be offline completely, but I expect to be checking for email rarely. The rest of the PL crew is at your service, of course! --Guido van Rossum (home page: http://www.python.org/~guido/) From gsw@agere.com Sun Oct 20 02:55:19 2002 From: gsw@agere.com (Gerald S. Williams) Date: Sat, 19 Oct 2002 21:55:19 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: Tim Peters wrote: > If you've actually got the energy to time these things , how about > adapting the int-mul overflow checking code in Python's intobject.c's > int_mul() function too? I'll take a look. > It could be simplified a little if it were specialized to unsigned mult. You're simply referring to breaking down the operation into "remove sign", multiply, and "generate sign" stages, right? I'll definitely keep it in mind. -Jerry Williams From tim.one@comcast.net Sun Oct 20 04:39:08 2002 From: tim.one@comcast.net (Tim Peters) Date: Sat, 19 Oct 2002 23:39:08 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: [Tim\ >> If you've actually got the energy to time these things , how >> about adapting the int-mul overflow checking code in Python's >> intobject.c's int_mul() function too? [Gerald S. Williams] > I'll take a look. Thanks! >> It could be simplified a little if it were specialized to unsigned >> mult. > You're simply referring to breaking down the operation into "remove > sign", multiply, and "generate sign" stages, right? I'll definitely > keep it in mind. Goodness no -- that kind of cleverness is usually buggy, and usually by failing to account for that the moral equivalent of -sys.maxint-1 is a fine product, but sys.maxint+1 isn't. I was talking specifically about the based-on-doubles int_mul(). In particular, the slow-path line const double absprod = doubleprod >= 0.0 ? doubleprod : -doubleprod; isn't needed if we know the product is >= 0. From skip@manatee.mojam.com Sun Oct 20 13:00:17 2002 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 20 Oct 2002 07:00:17 -0500 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200210201200.g9KC0H6L019011@manatee.mojam.com> Bug/Patch Summary ----------------- 316 open / 2963 total bugs (+4) 92 open / 1733 total patches (-4) New Bugs -------- textwrap fails on unicode using defaults (2002-10-13) http://python.org/sf/622831 inconsistent results of leading whitespace in textwrap input (2002-10-13) http://python.org/sf/622849 urllib.pathname2url(), .url2pathname() (2002-10-14) http://python.org/sf/623196 tempfile crashes (2002-10-15) http://python.org/sf/623464 __rdiv__ vs new-style classes (2002-10-15) http://python.org/sf/623669 Broken link in local documentation. (2002-10-16) http://python.org/sf/624024 email 2.4.3 pkg mail header error (2002-10-16) http://python.org/sf/624254 sq_concat prevents __radd__ from working (2002-10-17) http://python.org/sf/624807 Creation of struct_seq types (2002-10-17) http://python.org/sf/624827 help(UNICODE) -- reference page missing (2002-10-17) http://python.org/sf/624860 pydoc shows tools/scripts but blows up (2002-10-17) http://python.org/sf/624893 Typo at line 351 of email/Charset.py (2002-10-18) http://python.org/sf/625509 Errors with recursive objects (2002-10-19) http://python.org/sf/625698 Reorganize MacPython resources on OSX (2002-10-19) http://python.org/sf/625725 sys.executable in applets (2002-10-19) http://python.org/sf/625728 IDE starts in / on OSX (2002-10-19) http://python.org/sf/625734 missing names in telnetlib.py (2002-10-19) http://python.org/sf/625823 New Patches ----------- 2.2 patches for BSD/OS 5.0 (2002-03-26) http://python.org/sf/535335 autoconf-2.5X tweaks for configure.in (2002-10-15) http://python.org/sf/623771 HTTP Auth support for xmlrpclib (2002-10-16) http://python.org/sf/624180 attributes for urlsplit, urlparse result (2002-10-16) http://python.org/sf/624325 sets.BaseSet.isdisjointset(other) (2002-10-18) http://python.org/sf/625513 Closed Bugs ----------- Broken \ref link in documentation (2000-10-18) http://python.org/sf/217195 bad links at Ref Guide (2001-11-23) http://python.org/sf/484967 re.finditer (2002-07-24) http://python.org/sf/585882 list(xrange(sys.maxint / 4)) -> swapping (2002-09-28) http://python.org/sf/616019 segmentation fault importing huge source (2002-10-07) http://python.org/sf/619789 __ipow__ broken for 'object's (2002-10-08) http://python.org/sf/620179 core setup.py fails on Windows (2002-10-08) http://python.org/sf/620364 websucker relative-URL errors (2002-10-09) http://python.org/sf/620705 SocketServer/socket allow_reuse_address (2002-10-09) http://python.org/sf/620783 email: ASCII decoding error on 8bit data (2002-10-10) http://python.org/sf/621457 docs do not include spec info from PEPs (2002-10-10) http://python.org/sf/621526 Closed Patches -------------- specifying headers for extensions (2002-03-21) http://python.org/sf/533008 opt. timeouts for Queue.put() and .get() (2002-06-22) http://python.org/sf/572628 makesockaddr, use addrlen with AF_UNIX (2002-06-27) http://python.org/sf/574707 incompatible, but nice strings improveme (2002-07-08) http://python.org/sf/578688 smtplib.py patch for macmail esmtp auth (2002-07-17) http://python.org/sf/583180 db4 include not found (2002-08-02) http://python.org/sf/590377 zipfile.py reads archives with comments (2002-09-19) http://python.org/sf/611760 koi8_u codec (2002-09-23) http://python.org/sf/613173 tkFileDialog multiple option (2002-10-11) http://python.org/sf/621891 From python@rcn.com Mon Oct 21 04:03:04 2002 From: python@rcn.com (Raymond Hettinger) Date: Sun, 20 Oct 2002 23:03:04 -0400 Subject: [Python-Dev] __getitem__ in user defined classes Message-ID: <002001c278ae$60425e00$5bf7a4d8@oemcomputer> In Py2.3, __getitem__ conveniently supports slices for builtin sequences: 'abcde'.__getitem__(slice(2,4)) For user defined classes to emulate this behavior, they need to test the index argument to see whether it is a slice and then loop over the slice indices like this: class SquaresToTen: """Acts like a list of squares but computes only when needed""" def __len__(self): return 11 def __getitem__(self, index): if isinstance(index, slice): return [x**2 for x in range(index.start, index.stop, index.step)] else: return index**2 print SquaresToTen()[2] print SquaresToTen()[7:1:-2] This could be simplified somewhat by making slices iterable so that the __getitem__ definition looks more like this: def __getitem__(self, index): if isinstance(index, slice): return [x**2 for x in index] else: return index**2 Raymond Hettinger From Raymond Hettinger" I'm filling in while Brett is on vacation. To make everyone appreciate his formatting and lucidity, I'll skip the reStructuredText and write in my own terse style. http://mail.python.org/pipermail/python-dev/2002-October/029577.html Python 2.2.2. was released in final form this week. Oh, yeah! http://mail.python.org/pipermail/python-dev/2002-October/029527.html Why does the Window's installer use shortened 8.3 pathnames? Answer: It doesn't have to but it less confusing than dealing with pathnames that can have embedded spaces. http://mail.python.org/pipermail/python-dev/2002-October/029544.html PEP 218 was updated to match the sets.py module in CVS for Python 2.3. The question arose as to whether immutable sets should be implemented in C as a separate type or whether mutability should be a property. Answer: Keep them as different types since they support different methods. Next question: Should there be a separate language syntax for the two different types. Answer: Probably not, leave as an open question in the PEP. http://mail.python.org/pipermail/python-dev/2002-October/029542.html The -U flag was put in to test the Python standard lib for being able to handle Unicode seamlessly. It remained undocumented so that bug reports wouldn't pile up while the issues were being worked out. Progress is being made but it is possible the -U flag will never be fully implemented and released as a documented feature. http://mail.python.org/pipermail/python-dev/2002-October/029535.html Python has more than one place that is vulnerable to 32 bit word overflows unless a bunch of tests are put in place to catch unusual highly contrived cases that never come up in real world examples. Since the fixes would be ugly, wordy, and slow, it was decided to leave them alone. http://mail.python.org/pipermail/python-dev/2002-October/029024.html Tcl 8.0 and 8.1 are ancient history except in the world of Cygwin which is currently stuck with 8.0. Accordingly, Python 2.3's support for the prior versions has been cleaned out and support for UCS-4 in _tkinter has been added. http://mail.python.org/pipermail/python-dev/2002-October/029591.html Getting mtime, atime, and ctime to return floats instead of ints seemed like a simple enough change. Much discussion was held on how to accommodate the change without breaking anything that relied on the type. Though breakage would've been rare (since the only normal use for the data is to compare it or print it), Guido proposed a solution (putting the ints in the tuple) that seemed to accommodate all practical cases. http://mail.python.org/pipermail/python-dev/2002-October/029679.html Brian Quinlan teasesd out the design issue behind proxies and weakref objects. From tim.one@comcast.net Mon Oct 21 05:56:20 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 21 Oct 2002 00:56:20 -0400 Subject: [Python-Dev] Discussion Draft: PyDev Summary for week ending 20-Oct-2002 In-Reply-To: <00f601c278ba$62892340$5bf7a4d8@oemcomputer> Message-ID: [Raymond Hettinger] > I'm filling in while Brett is on vacation. To make everyone > appreciate his formatting and lucidity, I'll skip the > reStructuredText and write in my own terse style. It has its virtues! For example, I actually read this to the end . > ... > Python has more than one place that is vulnerable to 32 bit word > overflows unless a bunch of tests are put in place to catch unusual > highly contrived cases that never come up in real world examples. > Since the fixes would be ugly, wordy, and slow, it was decided to > leave them alone. That's just what I would do (i.e., leave them alone). Security Weenies will eat me for lunch because of it, and I don't think anyone else wants to join me at the center of the buffet. That is, this boondoggle had an Air of Inevitability to it. Curious: We're missing a section about your new implementation of the Mersenne Twister . Speaking of which, I haven't been able to get a copy, but Marsaglia (a Big Name in the field) has another new generator called "The Monster", supposedly described in a preprint, that's simpler than the Twister and has a much longer period (on the order of 10**8000; in contrast, Python's current generator has period ~10**12). By "simpler", I mean that if you read the paper by the Twister's authors, you walked away from it knowing less than when you started <0.6 wink>. From greg@cosc.canterbury.ac.nz Mon Oct 21 04:39:53 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Mon, 21 Oct 2002 16:39:53 +1300 (NZDT) Subject: [Python-Dev] __getitem__ in user defined classes In-Reply-To: <002001c278ae$60425e00$5bf7a4d8@oemcomputer> Message-ID: <200210210339.g9L3drw15018@kuku.cosc.canterbury.ac.nz> Raymond Hettinger : > This could be simplified somewhat by making slices iterable Hmmm... if slices were iterable, they'd behave a lot like xrange objects. Perhaps slice == xrange could be true in some future Python? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From oren-py-d@hishome.net Mon Oct 21 10:09:34 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Mon, 21 Oct 2002 05:09:34 -0400 Subject: [Python-Dev] __getitem__ in user defined classes In-Reply-To: <200210210339.g9L3drw15018@kuku.cosc.canterbury.ac.nz> References: <002001c278ae$60425e00$5bf7a4d8@oemcomputer> <200210210339.g9L3drw15018@kuku.cosc.canterbury.ac.nz> Message-ID: <20021021090934.GA99584@hishome.net> On Mon, Oct 21, 2002 at 04:39:53PM +1300, Greg Ewing wrote: > Raymond Hettinger : > > > This could be simplified somewhat by making slices iterable > > Hmmm... if slices were iterable, they'd behave a lot > like xrange objects. > > Perhaps > > slice == xrange > > could be true in some future Python? http://python.org/sf/575515 , rejected. Oren From mwh@python.net Mon Oct 21 11:06:46 2002 From: mwh@python.net (Michael Hudson) Date: 21 Oct 2002 11:06:46 +0100 Subject: [Python-Dev] __getitem__ in user defined classes In-Reply-To: "Raymond Hettinger"'s message of "Sun, 20 Oct 2002 23:03:04 -0400" References: <002001c278ae$60425e00$5bf7a4d8@oemcomputer> Message-ID: <2mlm4sm7gp.fsf@starship.python.net> "Raymond Hettinger" writes: > In Py2.3, __getitem__ conveniently supports slices for > builtin sequences: 'abcde'.__getitem__(slice(2,4)) > > For user defined classes to emulate this behavior, they need > to test the index argument to see whether it is a slice and then > loop over the slice indices like this: > > class SquaresToTen: > """Acts like a list of squares but > computes only when needed""" > > def __len__(self): > return 11 > > def __getitem__(self, index): > if isinstance(index, slice): > return [x**2 for x in range(index.start, index.stop, index.step)] You can spell that if isinstance(index, slice): return [x**2 for x in range(*index.indices(11))] and, as a bonus, it'll work more often (consider "SquaresToTen()[7::-2]", for example). > This could be simplified somewhat by making slices iterable so that > the __getitem__ definition looks more like this: > > def __getitem__(self, index): > if isinstance(index, slice): > return [x**2 for x in index] > else: > return index**2 However to make omitted slice places work, you need to pass in the length of the sequence, so I don't think this can fly. Cheers, M. -- We've had a lot of problems going from glibc 2.0 to glibc 2.1. People claim binary compatibility. Except for functions they don't like. -- Peter Van Eynde, comp.lang.lisp From gsw@agere.com Mon Oct 21 14:33:52 2002 From: gsw@agere.com (Gerald S. Williams) Date: Mon, 21 Oct 2002 09:33:52 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: Tim Peters wrote: > Goodness no -- that kind of cleverness is usually buggy, and usually by > failing to account for that the moral equivalent of -sys.maxint-1 is a fine > product, but sys.maxint+1 isn't. Agreed. Also, signed values needn't be 2's complement numbers, and the behavior of signed overflow isn't guaranteed (it could saturate, for instance). The use of doubles makes sense from a portability standpoint. Or you could break out the signs and use unsigned longs with preprocessor (and configure) checks, being careful about performance impact. :-) For example, you might need to do something equivalent to this: #if HAVE_LIMITS_H #define MAX_POSITIVE_LONG_VALUE (LONG_MAX) #if -(LONG_MAX) == LONG_MIN #define MAX_NEGATIVE_LONG_VALUE (LONG_MAX) #elif (-(LONG_MAX) - 1) == LONG_MIN #define MAX_NEGATIVE_LONG_VALUE ((LONG_MAX) + 1) #else #undef MAX_POSITIVE_LONG_VALUE ... -Jerry From tim.one@comcast.net Mon Oct 21 22:47:02 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 21 Oct 2002 17:47:02 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: [Gerald S. Williams] > Agreed. Also, signed values needn't be 2's complement numbers, Python assumes 2's comp in several places already. > and the behavior of signed overflow isn't guaranteed (it could > saturate, for instance). But it doesn't assume anything about what happens on overflow, for either doubles or for signed integral types. > The use of doubles makes sense from a portability standpoint. I thought you were going to time this ? > Or you could break out the signs and use unsigned longs with > preprocessor (and configure) checks, being careful about performance > impact. :-) Breaking out the signs means tests and branches. That's something the int_mult() code in Python tries to avoid. Unfortunately, there's a ton of tests and branches before we ever get to the multiplication, trying to guess whether this is really a "sequence * integer" case. But once we get to actually multiplying two ints, the fast path has only one test+branch (did it overflow?). From gsw@agere.com Mon Oct 21 23:09:29 2002 From: gsw@agere.com (Gerald S. Williams) Date: Mon, 21 Oct 2002 18:09:29 -0400 Subject: [Python-Dev] Memory size overflows In-Reply-To: Message-ID: Tim Peters wrote: > Python assumes 2's comp in several places already. Yeah, I just noticed that. > I thought you were going to time this ? I am. If you assume that, for the vast majority of multiplications, both multiplicands are in the range -sqrt(sys.maxint):sqrt(sys.maxint), the following gives a 15% performance improvement for me: unsigned long sa = (a < 0L); unsigned long sb = (b < 0L); unsigned long ua = sa ? -a : a; unsigned long ub = sb ? -b : b; if ((ua | ub) & OVERFLOW_POSSIBLE_MASK) { } If both numbers are evenly distributed from -sys.maxint:sys.maxint, this yields about a 3% slowdown, though. These numbers are just for the core multiply path, since I pulled it out in order to compare algorithms better. I'm looking into improving the rest, but it's difficult. -Jerry From gward@python.net Tue Oct 22 20:24:39 2002 From: gward@python.net (Greg Ward) Date: Tue, 22 Oct 2002 15:24:39 -0400 Subject: [Python-Dev] textwrap and unicode Message-ID: <20021022192439.GA18528@cthulhu.gerg.ca> Well, my ignorance of Unicode has finally bitten me -- someone filed a bug (#622831) against textwrap.py because it crashes when it attempts to wrap a Unicode string. Here are the problems that I am aware of: * textwrap assumes "whitespace" means "the characters in string.whitespace" * textwrap assumes "lowercase letter" means "the characters in string.lowercase" (heck, this only works in English) Can someone tell me what the proper way to do this is? Or just point me at the relevant documentation? I've scoured the online docs and *Python Essential Reference*, and I know more about the codes and unicodedata modules than I did before. But I still don't know how to replace all whitespace with space, or detect words that end with a lowercase letter. Thanks -- Greg -- Greg Ward http://www.gerg.ca/ I brought my BOWLING BALL -- and some DRUGS!! From gsw@agere.com Tue Oct 22 20:48:48 2002 From: gsw@agere.com (Gerald S. Williams) Date: Tue, 22 Oct 2002 15:48:48 -0400 Subject: [Python-Dev] Re: Memory size overflows In-Reply-To: <20021022160006.7706.54880.Mailman@mail.python.org> Message-ID: I didn't find any way to improve the actual overflow check, although if you entirely replace the "fast path" check with checks involving unsigned masking, you get some performance improvement. For a wide variety of input patterns, I get about an 18% speedup versus the core long multiply code, when modified as shown below: #define UL_LO_HI_BIT (((unsigned long)1) << (sizeof(unsigned long) * 4U)) #define UL_LO_MASK ((UL_LO_HI_BIT) - 1) #define UL_HI_MASK (~(UL_LO_MASK)) #define UL_HI_LO_BIT (((unsigned long)1) << ((sizeof(unsigned long) * 4U)-1)) #define UL_OVERFLOW_IMPOSSIBLE_MASK ((UL_HI_LO_BIT) - 1) #define UL_OVERFLOW_POSSIBLE_MASK (~(UL_OVERFLOW_IMPOSSIBLE_MASK)) long core_int_mul(long a, long b) { long longprod = a * b; unsigned long ma = a & UL_HI_MASK; if (ma == ((a < 0) ? UL_HI_MASK : 0)) { unsigned long mb = b & UL_OVERFLOW_POSSIBLE_MASK; if (mb == ((b < 0) ? UL_OVERFLOW_POSSIBLE_MASK : 0)) { return longprod; } } { double doubleprod = (double)a * (double)b; double doubled_longprod = (double)longprod; double diff = doubled_longprod - doubleprod; double absdiff = (diff >= 0.0) ? diff : -diff; double absprod = (doubleprod >= 0.0) ? doubleprod : -doubleprod; /* absdiff/absprod <= 1/32 iff 32 * absdiff <= absprod -- 5 good bits is "close enough" */ if (32.0 * absdiff <= absprod) { return longprod; } else { SIGNAL_AN_ERROR; } } } This version suffers no apparent degradation versus the existing implementation when fed sets of multiplicands evenly distributed over range(-sys.maxint, sys.maxint), and almost always shows an improvement. Shall I submit a patch? -Jerry From zooko@zooko.com Tue Oct 22 20:51:36 2002 From: zooko@zooko.com (Zooko) Date: Tue, 22 Oct 2002 15:51:36 -0400 Subject: [Python-Dev] pymalloc.supp (a valgrind suppressions file) Message-ID: # add the cmdline arg `--suppressions=thisfile' when invoking valgrind { PyObject_Free does tricky inside-a-page peeking, but it's really okay. Cond fun:PyObject_Free } { PyObject_Free does tricky inside-a-page peeking, but it's really okay. Value4 fun:PyObject_Free } { PyObject_Free does tricky inside-a-page peeking, but it's really okay. Addr4 fun:PyObject_Free } { PyObject_Realloc does tricky inside-a-page peeking, but it's really okay. Cond fun:PyObject_Realloc } { PyObject_Realloc does tricky inside-a-page peeking, but it's really okay. Value4 fun:PyObject_Realloc } { PyObject_Realloc does tricky inside-a-page peeking, but it's really okay. Addr4 fun:PyObject_Realloc } From martin@v.loewis.de Tue Oct 22 20:58:07 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 22 Oct 2002 21:58:07 +0200 Subject: [Python-Dev] textwrap and unicode In-Reply-To: <20021022192439.GA18528@cthulhu.gerg.ca> References: <20021022192439.GA18528@cthulhu.gerg.ca> Message-ID: Greg Ward writes: > Here are the problems that I am aware of: > > * textwrap assumes "whitespace" means "the characters in > string.whitespace" I don't know how precisely you want to formulate the property. If x is a Unicode letter, then x.isspace() tells you whether it is a space character (this property holds for all characters of the Zs category, and all characters that have a bidirectionality of WS, B, or S). > * textwrap assumes "lowercase letter" means "the characters in > string.lowercase" (heck, this only works in English) Works the same way: x.islower() tells you whether a character is lower-case (meaning it is in the Ll category). HTH, Martin From mal@lemburg.com Tue Oct 22 21:01:32 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 22 Oct 2002 22:01:32 +0200 Subject: [Python-Dev] textwrap and unicode References: <20021022192439.GA18528@cthulhu.gerg.ca> Message-ID: <3DB5AE9C.1030607@lemburg.com> Greg Ward wrote: > Well, my ignorance of Unicode has finally bitten me -- someone filed a > bug (#622831) against textwrap.py because it crashes when it attempts to > wrap a Unicode string. > > Here are the problems that I am aware of: > > * textwrap assumes "whitespace" means "the characters in > string.whitespace" It should use u.isspace() for this. You might also want to consider u.splitlines() for line breaking, since Unicode has a lot more line breaking characters than ASCII (which u.splitlines() knows about). > * textwrap assumes "lowercase letter" means "the characters in > string.lowercase" (heck, this only works in English) u.lower() will do the right thing for Unicode. > Can someone tell me what the proper way to do this is? Or just point me > at the relevant documentation? I've scoured the online docs and *Python > Essential Reference*, and I know more about the codes and unicodedata > modules than I did before. But I still don't know how to replace all > whitespace with space, or detect words that end with a lowercase letter. Hope that helps, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From fredrik@pythonware.com Tue Oct 22 21:04:25 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Tue, 22 Oct 2002 22:04:25 +0200 Subject: [Python-Dev] textwrap and unicode References: <20021022192439.GA18528@cthulhu.gerg.ca> Message-ID: <00a201c27a06$45299260$ced241d5@hagrid> greg wrote: > But I still don't know how to replace all whitespace with > space string.join(phrase.split(), " ") or re.sub("(?u)\s+", " ", phrase) not sure which one's faster; I suggest benchmarking. (if you want to preserve leading/trailing space with the split approach, use isspace on the start/end of phrase) > or detect words that end with a lowercase letter. word[-1].islower() From gward@python.net Tue Oct 22 21:24:08 2002 From: gward@python.net (Greg Ward) Date: Tue, 22 Oct 2002 16:24:08 -0400 Subject: [Python-Dev] textwrap and unicode In-Reply-To: References: <20021022192439.GA18528@cthulhu.gerg.ca> Message-ID: <20021022202408.GA18718@cthulhu.gerg.ca> On 22 October 2002, Martin v. Loewis said: > I don't know how precisely you want to formulate the property. If x is > a Unicode letter, then x.isspace() tells you whether it is a space > character (this property holds for all characters of the Zs category, > and all characters that have a bidirectionality of WS, B, or S). OK, then it's an implementation problem rather than a "you can't get there from here" problem. Good. The reason I need a list of "whitespace chars" is to convert all whitespace to spaces; I use string.maketrans() and s.translate() to do this efficiently: class TextWrapper: [...] whitespace_trans = string.maketrans(string.whitespace, ' ' * len(string.whitespace)) [...] def _munge_whitespace(self, text): """_munge_whitespace(text : string) -> string Munge whitespace in text: expand tabs and convert all other whitespace characters to spaces. Eg. " foo\tbar\n\nbaz" becomes " foo bar baz". """ if self.expand_tabs: text = text.expandtabs() if self.replace_whitespace: text = text.translate(self.whitespace_trans) return text (The rationale: having tabs and newlines in a paragraph about to be wrapped doesn't make any sense to me.) Ahh, OK, I'm starting to see the problem: there's nothing wrong with the translate() method of strings or unicode strings, but string.maketrans() doesn't generate a mapping that u''.translate() likes. Hmmmm. Right, now I've RTFD'd (read the fine docstring) for u''.translate(). Here's what I've got now: whitespace_trans = string.maketrans(string.whitespace, ' ' * len(string.whitespace)) unicode_whitespace_trans = {} for c in string.whitespace: unicode_whitespace_trans[ord(unicode(c))] = ord(u' ') [...] def _munge_whitespace (self, text): [...] if self.replace_whitespace: if isinstance(text, str): text = text.translate(self.whitespace_trans) elif isinstance(text, unicode): text = text.translate(self.unicode_whitespace_trans) That's ugly as hell, but it works. Is there a cleaner way? The other bit of ASCII/English prejudice hardcoded into textwrap.py is this regex: sentence_end_re = re.compile(r'[%s]' # lowercase letter r'[\.\!\?]' # sentence-ending punct. r'[\"\']?' # optional end-of-quote % string.lowercase) You may recall this from the kerfuffle over whether there should be two spaces after a sentence in fixed-width fonts. The feature is there, and off by default, in TextWrapper. I'm not so concerned about this -- I mean, this doesn't even work with German or French, never mind Hebrew or Chinese or Hindi. Apart from the narrow definition of "lowercase letter", it has English punctuation conventions hardcoded into it. But still, it seems *awfully* dumb in this day and age to hardcode string.lowercase into a regex that's meant to detect "lowercase letters". But I couldn't find a better way to do it when I wrote this code last spring. Is there one? Thanks! Greg -- Greg Ward http://www.gerg.ca/ OUR PLAN HAS FAILED STOP JOHN DENVER IS NOT TRULY DEAD STOP HE LIVES ON IN HIS MUSIC STOP PLEASE ADVISE FULL STOP From tim.one@comcast.net Tue Oct 22 21:43:46 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 22 Oct 2002 16:43:46 -0400 Subject: [Python-Dev] Re: Memory size overflows In-Reply-To: Message-ID: [Gerald S. Williams] > I didn't find any way to improve the actual overflow check, > although if you entirely replace the "fast path" check with > checks involving unsigned masking, you get some performance > improvement. For a wide variety of input patterns, I get > about an 18% speedup versus the core long multiply code, > when modified as shown below: Which platform? Which compiler? What was your test driver? Was this timing the mult code in isolation, or timing Python-level multiplies? Claims of small speedups are notoriously platform- and test-dependent. If it's a mixed bag across platforms, the risk of introducing a new bug would favor leaving things alone. In the absence of a clear correctness proof, a Python simulation program demonstrating correctness exhaustively in small bases would also be helpful. > ... > Shall I submit a patch? Sure, but also submit your timing harness so that people can measure the effects cross-platform and cross-compiler. From martin@v.loewis.de Tue Oct 22 22:17:52 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 22 Oct 2002 23:17:52 +0200 Subject: [Python-Dev] textwrap and unicode In-Reply-To: <20021022202408.GA18718@cthulhu.gerg.ca> References: <20021022192439.GA18528@cthulhu.gerg.ca> <20021022202408.GA18718@cthulhu.gerg.ca> Message-ID: Greg Ward writes: > if self.replace_whitespace: > text = text.translate(self.whitespace_trans) > return text > > (The rationale: having tabs and newlines in a paragraph about to be > wrapped doesn't make any sense to me.) Is it then really necessary to replace each of these characters, or would it be acceptable to replace sequences of them, as Fredrik proposes (.split,.join)? > for c in string.whitespace: > unicode_whitespace_trans[ord(unicode(c))] = ord(u' ') There are conceptually 5 times as many space characters in Unicode (NO-BREAK SPACE, THREE-PER-EM SPACE, OGHAM SPACE MARK, and whatnot), but it is probably safe to ignore them. The complete fragment would read for c in range(sys.maxunicode): if unichr(c).isspace(): unicode_whitespace_trans[c] = u' ' (which is somewhat time-consuming, so you could hard-code a larger list if you wanted to) > That's ugly as hell, but it works. Is there a cleaner way? You may want to time re.sub, perhaps to find that the speed decrease is acceptable: space = re.compile("\s") text = space.sub(" ", text) > The other bit of ASCII/English prejudice hardcoded into textwrap.py is > this regex: > > sentence_end_re = re.compile(r'[%s]' # lowercase letter > r'[\.\!\?]' # sentence-ending punct. > r'[\"\']?' # optional end-of-quote > % string.lowercase) For the issue at hand: this code does "work" with Unicode, right? I.e. it will give some result, even if confronted with funny characters? If so, I think you can ignore this for the moment. > But I couldn't find a better way to do it when I wrote this code > last spring. Is there one? I believe the right approach is to support more classes in SRE. This one would be covered if there was a [:lower:] class. Regards, Martin From Jack.Jansen@oratrix.com Tue Oct 22 22:46:15 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Tue, 22 Oct 2002 23:46:15 +0200 Subject: [Python-Dev] Precompiled headers Message-ID: Gcc on MacOSX has the ability to use precompiled headers, and I was toying with the idea of giving this a try for Python.h. Has anyone tried this, and is it worth the hassle? Also, are precompiled headers an Apple addition to gcc, or are they available on all platforms? I'm interested in trying this because when I started using precompiled headers for MacPython with the CodeWarrior compiler the build time for a complete Python tree want down by about 70%, so that's definitely worth a little effort... -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From barry@python.org Tue Oct 22 23:09:42 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 22 Oct 2002 18:09:42 -0400 Subject: [Python-Dev] Precompiled headers References: Message-ID: <15797.52390.563124.805066@gargle.gargle.HOWL> >>>>> "JJ" == Jack Jansen writes: JJ> Gcc on MacOSX has the ability to use precompiled headers, and JJ> I was toying with the idea of giving this a try for JJ> Python.h. Has anyone tried this, and is it worth the hassle? JJ> Also, are precompiled headers an Apple addition to gcc, or are JJ> they available on all platforms? I don't know, but didn't the old NeXTSTEP Objective-C compilers (also based off of gcc) support pre-compiled headers? -Barry From martin@v.loewis.de Tue Oct 22 23:54:11 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 23 Oct 2002 00:54:11 +0200 Subject: [Python-Dev] Precompiled headers In-Reply-To: References: Message-ID: Jack Jansen writes: > Also, are precompiled headers an Apple addition to gcc, or are they > available on all platforms? It's an Apple addition. Apple has a long-standing plan to contribute this back, but there are several things in the way; mainly that it is a large ugly hack. > I'm interested in trying this because when I started using precompiled > headers for MacPython with the CodeWarrior compiler the build time for > a complete Python tree want down by about 70%, so that's definitely > worth a little effort... I find the compile times of the Python tree on my Pentium 930MHz system to be accceptable, so I'd personally rather avoid a feature with the potential of producing incorrect binaries. Regards, Martin From Jack.Jansen@oratrix.com Wed Oct 23 01:42:29 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Wed, 23 Oct 2002 02:42:29 +0200 Subject: [Python-Dev] Precompiled headers In-Reply-To: Message-ID: <4F2AEB2A-E620-11D6-802C-003065517236@oratrix.com> On woensdag, oktober 23, 2002, at 12:54 , Martin v. Loewis wrote: > I find the compile times of the Python tree on my Pentium 930MHz > system to be accceptable, so I'd personally rather avoid a feature > with the potential of producing incorrect binaries. This last sentence sounds like you know something that I don't, could you explain? Precompiled headers keep track of the preprocessor symbols they depend on, and their value at precompile time, so I've always thought that this (together with correct Makefile dependencies) would be safe... -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From python@rcn.com Wed Oct 23 01:48:19 2002 From: python@rcn.com (Raymond Hettinger) Date: Tue, 22 Oct 2002 20:48:19 -0400 Subject: [Python-Dev] textwrap and unicode References: <20021022192439.GA18528@cthulhu.gerg.ca> Message-ID: <009901c27a2d$e263bf20$45f6a4d8@oemcomputer> > Can someone tell me what the proper way to do this is? Or just point me > at the relevant documentation? I've scoured the online docs and *Python > Essential Reference*, and I know more about the codes and unicodedata > modules than I did before. But I still don't know how to replace all > whitespace with space, or detect words that end with a lowercase letter. Hang tight. I'm writing isspace() and islower() for UniCode. Then your fix will be much simpler and will leave the Uni-magic all in one place. Raymond Hettinger From Anthony Baxter Wed Oct 23 02:50:32 2002 From: Anthony Baxter (Anthony Baxter) Date: Wed, 23 Oct 2002 11:50:32 +1000 Subject: [Python-Dev] Precompiled headers In-Reply-To: Message-ID: <200210230150.g9N1oXn25708@localhost.localdomain> >>> Martin v. Loewis wrote > Jack Jansen writes: > > > Also, are precompiled headers an Apple addition to gcc, or are they > > available on all platforms? > > It's an Apple addition. Apple has a long-standing plan to contribute > this back, but there are several things in the way; mainly that it is > a large ugly hack. If it's anything like the way the old Lucid C++ compiler did it, it'll be just coredumping the preprocessor's state. We always found that it wasn't worthwhile unless you were dealing with something like OI's 1.something megabytes of headers... Anthony -- Anthony Baxter It's never too late to have a happy childhood. From bfordham@socialistsushi.com Wed Oct 23 03:55:54 2002 From: bfordham@socialistsushi.com (Bryan L. Fordham) Date: Tue, 22 Oct 2002 22:55:54 -0400 (EDT) Subject: [Python-Dev] tamper-evident logs Message-ID: <20021022225332.P88383-100000@the.whole.net> here's a question for y'all: I've written a simple http proxy that logs the domains visited. The idea is that it keeps a list of the unique domains for each day. No problem. The problem is that I want it to be obvious if someone has deleted something from the logs. This is a little bit o' software meant to run on the user's box. It does not have to be impossible to tamper with it, nor do I need to be able to say what was removed; just "hey, someone futzed with the log!" Any ideas would be greatly appreciated. --B From fredrik@pythonware.com Wed Oct 23 06:58:31 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 23 Oct 2002 07:58:31 +0200 Subject: [Python-Dev] tamper-evident logs References: <20021022225332.P88383-100000@the.whole.net> Message-ID: <01cb01c27a59$39639220$ced241d5@hagrid> Bryan L. Fordham wrote: > I've written a simple http proxy that logs the domains visited. The idea > is that it keeps a list of the unique domains for each day. No problem. > The problem is that I want it to be obvious if someone has deleted > something from the logs. > > This is a little bit o' software meant to run on the user's box. It does > not have to be impossible to tamper with it, nor do I need to be able to > say what was removed; just "hey, someone futzed with the log!" > > Any ideas would be greatly appreciated. This mailing list is for development of Python itself, not for general python questions. I suggest posting to comp.lang.python, or maybe to a security forum. From martin@v.loewis.de Wed Oct 23 07:16:54 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 23 Oct 2002 08:16:54 +0200 Subject: [Python-Dev] Precompiled headers In-Reply-To: <4F2AEB2A-E620-11D6-802C-003065517236@oratrix.com> References: <4F2AEB2A-E620-11D6-802C-003065517236@oratrix.com> Message-ID: Jack Jansen writes: > On woensdag, oktober 23, 2002, at 12:54 , Martin v. Loewis wrote: > > I find the compile times of the Python tree on my Pentium 930MHz > > system to be accceptable, so I'd personally rather avoid a feature > > with the potential of producing incorrect binaries. > > This last sentence sounds like you know something that I don't, could > you explain? Precompiled headers keep track of the preprocessor > symbols they depend on, and their value at precompile time, so I've > always thought that this (together with correct Makefile dependencies) > would be safe... ... assuming it is correctly implemented. All implementations of precompiled headers I've used so far had certain limitations (sometimes documented, sometimes undocumented), which would cause users to throw away the precompiled stuff from time to time to make sure that the compiler would really pick up their source code. I don't know the Apple gcc implementation of precompiled headers, though - it might be flawless. Regards, Martin From martin@v.loewis.de Wed Oct 23 07:19:08 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 23 Oct 2002 08:19:08 +0200 Subject: [Python-Dev] textwrap and unicode In-Reply-To: <009901c27a2d$e263bf20$45f6a4d8@oemcomputer> References: <20021022192439.GA18528@cthulhu.gerg.ca> <009901c27a2d$e263bf20$45f6a4d8@oemcomputer> Message-ID: "Raymond Hettinger" writes: > I'm writing isspace() and islower() for UniCode. Ah, the time machine! Regards, Martin From mal@lemburg.com Wed Oct 23 09:12:48 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 23 Oct 2002 10:12:48 +0200 Subject: [Python-Dev] textwrap and unicode References: <20021022192439.GA18528@cthulhu.gerg.ca> <20021022202408.GA18718@cthulhu.gerg.ca> Message-ID: <3DB65A00.8010000@lemburg.com> Greg Ward wrote: > On 22 October 2002, Martin v. Loewis said: > > OK, then it's an implementation problem rather than a "you can't get > there from here" problem. Good. The reason I need a list of > "whitespace chars" is to convert all whitespace to spaces; I use > string.maketrans() and s.translate() to do this efficiently: Use the trick Fredrik posted: u' '.join(x.split()) (.split() defaults to splitting on whitespace, Unicode whitespace if x is Unicode). > Ahh, OK, I'm starting to see the problem: there's nothing wrong with the > translate() method of strings or unicode strings, but string.maketrans() > doesn't generate a mapping that u''.translate() likes. Hmmmm. Unicode uses a different API for this since it wouldn't make sense to pass a sys.maxunicode character Unicode string to translate just to map a few characters. > The other bit of ASCII/English prejudice hardcoded into textwrap.py is > this regex: > > sentence_end_re = re.compile(r'[%s]' # lowercase letter > r'[\.\!\?]' # sentence-ending punct. > r'[\"\']?' # optional end-of-quote > % string.lowercase) > > You may recall this from the kerfuffle over whether there should be two > spaces after a sentence in fixed-width fonts. The feature is there, and > off by default, in TextWrapper. I'm not so concerned about this -- I > mean, this doesn't even work with German or French, never mind Hebrew or > Chinese or Hindi. Apart from the narrow definition of "lowercase > letter", it has English punctuation conventions hardcoded into it. But > still, it seems *awfully* dumb in this day and age to hardcode > string.lowercase into a regex that's meant to detect "lowercase > letters". But I couldn't find a better way to do it when I wrote this > code last spring. Is there one? There are far too many lowercase characters in Unicode to make this approach usable. It would be better if there were a way to use Unicode character categories in the re sets. Since that's not available, why not search for all potential sentence ends and then try all of the using .islower() in a for-loop ?! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From bfordham@socialistsushi.com Wed Oct 23 14:07:44 2002 From: bfordham@socialistsushi.com (Bryan L. Fordham) Date: Wed, 23 Oct 2002 09:07:44 -0400 Subject: [Python-Dev] tamper-evident logs References: <20021022225332.P88383-100000@the.whole.net> <01cb01c27a59$39639220$ced241d5@hagrid> Message-ID: <3DB69F20.1070900@socialistsushi.com> Fredrik Lundh wrote >This mailing list is for development of Python itself, not for general >python questions. I suggest posting to comp.lang.python, or maybe >to a security forum. > > yeah yeah yeah.... 8) Seriously, I just clicked the wrong address when typing the email. Sorry. Please don't send anyone for me in the middle of the night (or the middle of the day, for that matter)... My apologies. --B From radix@twistedmatrix.com Wed Oct 23 18:32:47 2002 From: radix@twistedmatrix.com (Christopher Armstrong) Date: Wed, 23 Oct 2002 19:32:47 +0200 Subject: [Python-Dev] src/Mac/Tools/IDE/PyFontify.py Message-ID: <20021023173247.GU12550@twistedmatrix.com> Hi! Python2.3 contains, inside the src/Mac/Tools/IDE/ directory a perfectly portable module called "PyFontify". Can this be added to the Python standard library somewhere under site-packages/? Twisted Matrix Labs distributes it with Twisted manually (with the copyright intact, of course), but this isn't very nice. -- Christopher Armstrong << radix@twistedmatrix.com >> http://twistedmatrix.com/users/radix.twistd/ From fredrik@pythonware.com Wed Oct 23 21:03:00 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 23 Oct 2002 22:03:00 +0200 Subject: [Python-Dev] src/Mac/Tools/IDE/PyFontify.py References: <20021023173247.GU12550@twistedmatrix.com> Message-ID: <00c701c27acf$322d6550$ced241d5@hagrid> Christopher Armstrong wrote: > Python2.3 contains, inside the src/Mac/Tools/IDE/ directory a > perfectly portable module called "PyFontify". Can this be added to the > Python standard library somewhere under site-packages/? Twisted Matrix > Labs distributes it with Twisted manually (with the copyright intact, > of course), but this isn't very nice. any reason you cannot just use the standard "tokenize" module? From just@letterror.com Wed Oct 23 21:08:51 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 23 Oct 2002 22:08:51 +0200 Subject: [Python-Dev] src/Mac/Tools/IDE/PyFontify.py In-Reply-To: <00c701c27acf$322d6550$ced241d5@hagrid> Message-ID: Fredrik Lundh wrote: > any reason you cannot just use the standard "tokenize" module? Hear hear. PyFontify.py is dead. And it sucks. Just From pobrien@orbtech.com Wed Oct 23 23:22:28 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Wed, 23 Oct 2002 17:22:28 -0500 Subject: [Python-Dev] OT: Pickling article Message-ID: <200210231722.28376.pobrien@orbtech.com> I recently finished an article for IBM's developerWorks website on Python= =20 Persistence Management. I basically cover quite a bit of material on=20 pickling/unpickling. I'm looking for a couple of pickling experts to crit= ique=20 the article before it gets published. If you'd like to do so, please cont= act=20 me offlist at pobrien@orbtech.com. Thanks. --=20 Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From mal@lemburg.com Thu Oct 24 09:50:42 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 24 Oct 2002 10:50:42 +0200 Subject: [Python-Dev] Re: Moving to Unicode 3.2 References: <3DB13012.3040604@lemburg.com> Message-ID: <3DB7B462.8080702@lemburg.com> Martin v. Loewis wrote: > "M.-A. Lemburg" writes: > > >>I haven't seen any messages about this on python-dev. Did I miss >>something ? > > No. For a change like this, I did not think consultation was > necessary. I saw that :-) >>The switch from Unicode 3.0 is a big one since 3.2 introduces >>non-BMP character points for the first time. > > I disagree; it's a small change. Just look at the patch itself: apart > from the (considerably large) generated data, there were very few > actual changes to source code: Changing a few limits was sufficient. Which underlines Fredrik's good design of the Unicode database. Still, the changes from Unicode 3.0 to 3.2 are significant (to the few users who actually make use of the database): http://www.unicode.org/unicode/reports/tr28/ Looking at a diff of the 3.0 and the 3.2 data files you can find quite a few additions, changes in categories and several removals of numeric interpretations of code points. Most important is, of course, that 3.2 actually assing code points outside the 16-bit Unicode range. What I'm really concerned about is that Python is moving on the dev edge here while most other technologies are still using Unicode 2.0. It's nice to be able to use Python as reference implementation for Unicode, but the interoperability between Python and Java/Windows suffers from this. > Since there are no backwards compatibility issues, and no design > choices (apart from the choice of updating the database at all), this > is a straight-forward change. > > >>I also don't think that it is a good idea to ship the Unicode >>3.2 database while the code behaves as defined in Unicode 3.0. > > Can you please elaborate? What code behaves as defined in Unicode 3.0 > that is incompatible with the Unicode 3.2 database? Well, I should have written: ... while the code does not even fully implement Unicode 3.0. Fortunately, you have already started working in that direction (adding normalization) and I'd like to thank you for your efforts. >>And last not least, I'd like to be asked before you make such >>changes. > > > I find this quite a possessive view, and I would prefer if you bring > up technical arguments instead of procedural ones, but ok... I am not trying to possess anything here. This is about managing the Unicode code base and I'm still under the impression that I'm the one in charge here. > I was under the impression that I can apply my own professional > judgement when deciding what patches to apply without consultation, in > what cases to ask on python-dev, and when to submit a patch to SF. > > Apparently, this impression is wrong. Can you please give precise > instructions what constitutes "such a change"? I consider changes in the design or additions that affect the design such a change. > Also, should I back this change out? No, let's first find out what the consequences of this change are and then decide whether it's a good idea or not. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Thu Oct 24 13:34:25 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 24 Oct 2002 14:34:25 +0200 Subject: [Python-Dev] Re: Moving to Unicode 3.2 In-Reply-To: <3DB7B462.8080702@lemburg.com> References: <3DB13012.3040604@lemburg.com> <3DB7B462.8080702@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > Still, the changes from Unicode 3.0 to 3.2 are significant (to the > few users who actually make use of the database): > > http://www.unicode.org/unicode/reports/tr28/ > > Looking at a diff of the 3.0 and the 3.2 data files you can > find quite a few additions, changes in categories and several > removals of numeric interpretations of code points. Most > important is, of course, that 3.2 actually assing code points > outside the 16-bit Unicode range. I still can't see why adding code points beyond the BMP is most important part of this change. If anything, the changes to categories might affect applications. In all these cases, I'll assume that the more recent database is more correct, so any change in behaviour is likely for the better. > What I'm really concerned about is that Python is moving on the dev > edge here while most other technologies are still using Unicode > 2.0. It's nice to be able to use Python as reference implementation > for Unicode, but the interoperability between Python and > Java/Windows suffers from this. What do you mean "most other technologies"? Windows 2000+ support UTF-16 at the API quite happily, OpenType supports UCS-4, XML is defined in terms of Unicode-including-extended-planes, Unicode normalization (mandated by all W3C recommendations) is defined in terms of Unicode 3.1, the character database of JDK 1.4 is based on Unicode 3.0. I don't think there are that many technologies which still use Unicode 2.0 (explicitly: most applications don't actually care whether a code point is assigned or not). > Well, I should have written: ... while the code does not even > fully implement Unicode 3.0. > > Fortunately, you have already started working in that > direction (adding normalization) and I'd like to thank > you for your efforts. While looking at normalization, I noticed an aspect that still confuses me: When new characters are added to Unicode, you can get additional canonical decompositions. Applying the composition algorithm mechanically, that could cause certain normal forms to change depending on the Unicode version. Therefore, TR#15 defines a composition version of Unicode to be used in normalization, see http://www.unicode.org/unicode/reports/tr15/#Versioning This version is defined to be Unicode 3.1.0. In turn, it seems that algorithms that perform normalization based on older databases are incorrect (algorithms based on newer databases must take the exclusion tables into account, which my patch does) [this is the confusing part: how wrong are normalization algorithms based on, say, Unicode 3.0?] So for this specific aspect, updating the Unicode database was really necessary. > > Apparently, this impression is wrong. Can you please give precise > > instructions what constitutes "such a change"? > > I consider changes in the design or additions that affect > the design such a change. That is still unclear. What's an addition? I did not really add anything - I changed what was already there. Nor do I think I changed the design. Regards, Martin From radix@twistedmatrix.com Thu Oct 24 18:54:21 2002 From: radix@twistedmatrix.com (Christopher Armstrong) Date: Thu, 24 Oct 2002 19:54:21 +0200 Subject: [Python-Dev] src/Mac/Tools/IDE/PyFontify.py In-Reply-To: References: <00c701c27acf$322d6550$ced241d5@hagrid> Message-ID: <20021024175420.GB631@twistedmatrix.com> On Wed, Oct 23, 2002 at 10:08:51PM +0200, Just van Rossum wrote: > Fredrik Lundh wrote: > > > any reason you cannot just use the standard "tokenize" module? > > Hear hear. PyFontify.py is dead. And it sucks. Well, the reason we used it was that py2html used it. :-) Anyway, we searched for alternatives, and only came across Norvig's py2html which is very broken, so Moshe Zadka wrote a new htmlizer in half an hour today, based on tokenizer. Hooray NIH! twisted.python.htmlizer. ;) -- Christopher Armstrong << radix@twistedmatrix.com >> http://twistedmatrix.com/users/radix.twistd/ From dave@boost-consulting.com Thu Oct 24 23:41:04 2002 From: dave@boost-consulting.com (David Abrahams) Date: 24 Oct 2002 18:41:04 -0400 Subject: [Python-Dev] pyconfig.h and exec-prefix Message-ID: This issue came up on the c++-sig. I'm not qualified to answer any of the questions at the bottom, and hoped that this forum could be of some help. From: "Paul F. Kunz" Subject: Re: [C++-sig] Boost V1 build on Solaris To: c++-sig@python.org Date: Wed, 23 Oct 2002 17:36:58 -0700 Reply-To: c++-sig@python.org One final question, which is not specific to boost::python but general question on building Python modules. What my Solaris administrators have done is to configure Python with > ./configure --prefix=/afs/.slac.stanford.edu/package/python/common/2.0 --exec-prefix=/afs/.slac.stanford.edu/package/python/sun4x_55/2.0 They also did a build fron Linux with the appropriate `exec-prefix'. What apparently this has done is to put all but one of the Python include files in the directory `common/2.0/include'. But Python.h includes `config.h' or `pyconfig.h' (depending on the version) which gets installed in `sun4x_55/2.0/include'. This is appropriate since this file is machine/OS dependent. Now the problem is that boost::python, and other packages, gives you one variable, like PYTHON_INCLUDES, for its build. When it is set to find `Python.h' it will not find `pyconfig.h'. What is the best way to handle this situation? Is it a bug in Python's configure script? Bug in boost::python and others? Or am I missing something obvious? -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Building C/C++ Extensions for Python: Dec 9-11, Austin, TX http://www.enthought.com/training/building_extensions.html From skip@pobox.com Fri Oct 25 00:11:11 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 24 Oct 2002 18:11:11 -0500 Subject: [Python-Dev] pyconfig.h and exec-prefix In-Reply-To: References: Message-ID: <15800.32271.195128.852080@montanaro.dyndns.org> >> Now the problem is that boost::python, and other packages, gives you >> one variable, like PYTHON_INCLUDES, for its build. When it is set to >> find `Python.h' it will not find `pyconfig.h'. Seems to me like boost::python needs to be able to point at both directories. Maybe PYTHON_INCLUDES could be a colon-separated list of directories which gets broken up into multiple -I flags. I suppose that sort of change is a bit late for v1? Skip From dave@boost-consulting.com Fri Oct 25 00:01:48 2002 From: dave@boost-consulting.com (David Abrahams) Date: 24 Oct 2002 19:01:48 -0400 Subject: [Python-Dev] pyconfig.h and exec-prefix In-Reply-To: <15800.32271.195128.852080@montanaro.dyndns.org> References: <15800.32271.195128.852080@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > >> Now the problem is that boost::python, and other packages, gives you > >> one variable, like PYTHON_INCLUDES, for its build. When it is set to > >> find `Python.h' it will not find `pyconfig.h'. > > Seems to me like boost::python needs to be able to point at both > directories. Maybe PYTHON_INCLUDES could be a colon-separated list of > directories which gets broken up into multiple -I flags. I suppose > that sort of change is a bit late for v1? Considering that v2 was released a few weeks ago, that sounds about right ;-) -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Building C/C++ Extensions for Python: Dec 9-11, Austin, TX http://www.enthought.com/training/building_extensions.html From skip@pobox.com Fri Oct 25 00:59:27 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 24 Oct 2002 18:59:27 -0500 Subject: [Python-Dev] pyconfig.h and exec-prefix In-Reply-To: References: <15800.32271.195128.852080@montanaro.dyndns.org> Message-ID: <15800.35167.82866.831754@montanaro.dyndns.org> >> Seems to me like boost::python needs to be able to point at both >> directories. Maybe PYTHON_INCLUDES could be a colon-separated list >> of directories which gets broken up into multiple -I flags. I >> suppose that sort of change is a bit late for v1? David> Considering that v2 was released a few weeks ago, that sounds David> about right ;-) How about OTHER= Dear Python developers, I have a question about "dict" subclasses and eval() that I would kindly like to ask. The problem I have is the following: For a web content system in Python, I am using Python's eval() function to run code embedded in templates. Currently, the code failes with a NameError if a variable is referenced inside the code that is not defined in the locals or globals dictionary. Unfortunately, this is a rather common situation: one often uses variables (like error codes) that aren't set all the time. Intializing these variables is a tedious job so a default value would suit me nicely. I created a derived class of the standard `dict' that fills in a default value when a key is not found. This is exactly the same as Guido describes in his "descintro" paper. I tried to use this dictionary as the "globals" parameter with eval(). As Guido already describes in his paper, this doesn't work. Quoting Guido: "The interpreter uses an internal function to access the dictionary, which bypasses our __getitem__() override. I admit that this can be a problem (although it is only a problem in this context, when a dict subclass is used as a locals/globals dictionary); it remains to be seen if I can fix this without compromising performance in the common case." Is there a solution to this problem in sight? Or altenately, is there a way I can find out which variables are used inside a compiled code block so I can initialize non-specified variables? I have a vague memory that the nested scopes feature has to determine at compile time which variables are being used in a code block. Thanks for your time! Greetings, Geert Jansen From martin@v.loewis.de Fri Oct 25 17:50:02 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 25 Oct 2002 18:50:02 +0200 Subject: [Python-Dev] Using defaultdict as globals/locals for eval() In-Reply-To: <20021025162651.GA9056@vader.boskant.nl> References: <20021025162651.GA9056@vader.boskant.nl> Message-ID: Geert Jansen writes: > Is there a solution to this problem in sight? No, although contributions are welcome. > Or altenately, is there a way I can find out which variables are > used inside a compiled code block so I can initialize non-specified > variables? I have a vague memory that the nested scopes feature has > to determine at compile time which variables are being used in a > code block. Sure: you can parse the code, build an AST tuple (or list), and traverse that. This, of course, is off-topic for python-dev. Regards, Martin From nas@python.ca Fri Oct 25 18:01:29 2002 From: nas@python.ca (Neil Schemenauer) Date: Fri, 25 Oct 2002 10:01:29 -0700 Subject: [Python-Dev] Using defaultdict as globals/locals for eval() In-Reply-To: References: <20021025162651.GA9056@vader.boskant.nl> Message-ID: <20021025170129.GA18253@glacier.arctrix.com> Martin v. Loewis wrote: > Geert Jansen writes: > > Or altenately, is there a way I can find out which variables are > > used inside a compiled code block so I can initialize non-specified > > variables? I have a vague memory that the nested scopes feature has > > to determine at compile time which variables are being used in a > > code block. > > Sure: you can parse the code, build an AST tuple (or list), and > traverse that. You might want to look at the ptl_compile module in Quixote for some ideas. The AST from the 'compiler' package is much nicer than the parse tree from the 'parser' module. Neil From gsw@agere.com Fri Oct 25 18:07:14 2002 From: gsw@agere.com (Gerald S. Williams) Date: Fri, 25 Oct 2002 13:07:14 -0400 Subject: [Python-Dev] Re: Memory size overflows In-Reply-To: Message-ID: I wrote: > Shall I submit a patch? Tim Peters wrote: > Sure, but also submit your timing harness so that people can measure the > effects cross-platform and cross-compiler. Of course. I first wanted to see if there was any interest. Integrated into Python, the speedup was much less: only 0.1% overall for this million-multiply script: import time start = time.time() a = -1000 while a <= 1000: b = 1000 while b <= 1000: b += 1 a += 1 end = time.time() base = end - start start = time.time() a = -1000 while a <= 1000: b = -1000 while b <= 1000: b += 1 c = a * b a += 1 end = time.time() print end - start - base The difference in speedup is an interesting data point on Python overhead in general (it took just over 6 seconds to do 1,000,000 multiplies on my system). > Which platform? Which compiler? What was your test driver? Was this > timing the mult code in isolation, or timing Python-level multiplies? > Claims of small speedups are notoriously platform- and test-dependent. If > it's a mixed bag across platforms, the risk of introducing a new bug would > favor leaving things alone. In the absence of a clear correctness proof, a > Python simulation program demonstrating correctness exhaustively in small > bases would also be helpful. I doubt you're still interested, but I used Cygwin on a P3/833 Thinkpad running Win2K (GCC compiler). My original results were on just the multiply code in isolation--I wanted to see if there was interest before continuing. Since I have them, I attached my initial testbed (new.c is the final version), and here is an exhaustive test for the "cannot_overflow" check used in that version: from __future__ import generators def cannot_overflow(a,b,m): M_MASK = (1 << m) - 1 HI_LO_BIT = 1 << ((m/2) - 1) OVL_IMPOSSIBLE_MASK = HI_LO_BIT - 1 OVL_POSSIBLE_MASK = M_MASK ^ OVL_IMPOSSIBLE_MASK if a < 0: check = OVL_POSSIBLE_MASK else: check = 0 if (a & OVL_POSSIBLE_MASK) == check: if b < 0: check = OVL_POSSIBLE_MASK else: check = 0 if (b & OVL_POSSIBLE_MASK) == check: return True return False def overflowed(a,b,m): VALID_MASK = (2**(m - 1)) - 1 INVALID_MASK = (2**(2*m) - 1) ^ VALID_MASK prod = a * b highbits = prod & INVALID_MASK return (highbits != 0) and (highbits != INVALID_MASK) def checkall(m): for i in range(-(2**(m-1)),2**(m-1)): for j in range(-(2**(m-1)),2**(m-1)): if cannot_overflow(i,j,m) and overflowed(i,j,m): yield (i,j) if __name__ == "__main__": for m in range(2,11,2): print "M ==", m for i in checkall(m): print i print "END OF LIST" You can compare the "cannot_overflow" code to the code that would be added to int_mul. For the sake of minimizing impact, I limited the changes to defining UL_OVERFLOW_POSSIBLE_MASK as the equivalent of ~(2**(m/2-1) - 1) and this code, added immediately after converting the arguments into longs: { unsigned long ma = a & UL_OVERFLOW_POSSIBLE_MASK; if (ma == ((a < 0) ? UL_OVERFLOW_POSSIBLE_MASK : 0)) { unsigned long mb = b & UL_OVERFLOW_POSSIBLE_MASK; if (mb == ((b < 0) ? UL_OVERFLOW_POSSIBLE_MASK : 0)) { return PyInt_FromLong(a*b); } } } This seems pretty safe, but the payout seems too small to warrant taking this any further. Unless someone asks me to do so, I'm planning to drop it. -Jerry From alex_c@MIT.EDU Sat Oct 26 21:34:39 2002 From: alex_c@MIT.EDU (Alex Coventry) Date: Sat, 26 Oct 2002 16:34:39 -0400 (EDT) Subject: [Python-Dev] "conflicting types" in python CVS build Message-ID: <200210262034.QAA15149@multics.mit.edu> hi. i built a fresh checkout of python from CVS, and got this error: Parser/pgenmain.c:169: conflicting types for PyOS_Readline' Include/pythonrun.h:116: previous declaration of PyOS_Readline' make: *** [Parser/pgenmain.o] Error 1 cvs annoate says the latest prototype for PyOS_Readline is in pythonrun.h. Commenting that line out results in a complete build that passes the test suite. hope this helps. alex. From martin@v.loewis.de Sun Oct 27 09:23:29 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 27 Oct 2002 10:23:29 +0100 Subject: [Python-Dev] "conflicting types" in python CVS build In-Reply-To: <200210262034.QAA15149@multics.mit.edu> References: <200210262034.QAA15149@multics.mit.edu> Message-ID: Alex Coventry writes: > hi. i built a fresh checkout of python from CVS, and got this error: > > Parser/pgenmain.c:169: conflicting types for PyOS_Readline' > Include/pythonrun.h:116: previous declaration of PyOS_Readline' > make: *** [Parser/pgenmain.o] Error 1 > > cvs annoate says the latest prototype for PyOS_Readline is in > pythonrun.h. Commenting that line out results in a complete build that > passes the test suite. > > hope this helps. Not really. Can you please copy line 169 of pgenmain.c? I can't see a problem. Regards, Martin From skip@manatee.mojam.com Sun Oct 27 13:00:39 2002 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 27 Oct 2002 07:00:39 -0600 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200210271300.g9RD0dev003909@manatee.mojam.com> Bug/Patch Summary ----------------- 318 open / 2981 total bugs (no change) 96 open / 1743 total patches (+4) New Bugs -------- email incompatibility upgrading to 2.2.2 (2002-10-20) http://python.org/sf/626119 missing DECREF's in embedding example (2002-10-21) http://python.org/sf/626275 Support RFC 2111 in email package (2002-10-21) http://python.org/sf/626452 urllib2 doesn't do HTTP-EQUIV & Refresh (2002-10-21) http://python.org/sf/626543 strptime() always returns 0 in dst field (2002-10-21) http://python.org/sf/626570 Build error using make VPATH feature (2002-10-22) http://python.org/sf/626926 canvas.create_box() crashes Tk thread (2002-10-22) http://python.org/sf/626936 KeyPress bindings don't work (2002-10-23) http://python.org/sf/627798 curses library not found during make (2002-10-23) http://python.org/sf/627864 Set constructor fails with NameError (2002-10-24) http://python.org/sf/628246 pydoc.Helper.topics not based on docs (2002-10-24) http://python.org/sf/628258 Deprecate PyNumber_Check (2002-10-25) http://python.org/sf/628842 pickle won't dump instances after reload (2002-10-25) http://python.org/sf/628925 Race condition in asyncore poll (2002-10-26) http://python.org/sf/629097 build does not respect --prefix (2002-10-27) http://python.org/sf/629345 New Patches ----------- db4 include not found (2002-08-02) http://python.org/sf/590377 Support Unicode normalization (2002-10-21) http://python.org/sf/626485 Support Hangul Syllable names (2002-10-21) http://python.org/sf/626548 Bytecode copy bug in freeze (2002-10-24) http://python.org/sf/627900 xmlrpclib: Optional 'nil' support (2002-10-24) http://python.org/sf/628208 Experimental Inno Setup Win32 installer (2002-10-24) http://python.org/sf/628301 Detect BLT on more systems (2002-10-26) http://python.org/sf/629126 install lib-dynload .so files mode 555 (2002-10-26) http://python.org/sf/629278 Closed Bugs ----------- mac module documentation inaccuracy. (2002-01-17) http://python.org/sf/505150 ConfigParser bug/limitation (2002-02-01) http://python.org/sf/511737 ConfigParser cleanup, items() [needs tests] (2002-04-17) http://python.org/sf/545096 Nested anchors in Ref Man index.html (2002-04-20) http://python.org/sf/546579 Wrong description for PyErr_Restore (2002-07-03) http://python.org/sf/577000 PyString_AsString underdocumented (2002-09-08) http://python.org/sf/606463 packages doc in the tutorial (2002-09-29) http://python.org/sf/616211 PythonLauncher Dialogs do not work (2002-10-06) http://python.org/sf/619245 test_builtin hangs on MacOSX 10.2.1 (2002-10-06) http://python.org/sf/619400 sizeof(struct stat) changes (2002-10-10) http://python.org/sf/621523 urllib.pathname2url(), .url2pathname() (2002-10-14) http://python.org/sf/623196 Typo at line 351 of email/Charset.py (2002-10-18) http://python.org/sf/625509 IDE starts in / on OSX (2002-10-19) http://python.org/sf/625734 Closed Patches -------------- readline /dev/tty problem (2002-02-04) http://python.org/sf/512981 Installation database patch (2002-05-29) http://python.org/sf/562100 add unescape method to xml.sax.saxutils (2002-09-23) http://python.org/sf/613256 autoconf-2.5X tweaks for configure.in (2002-10-15) http://python.org/sf/623771 From alex_c@MIT.EDU Sun Oct 27 13:47:28 2002 From: alex_c@MIT.EDU (Alex Coventry) Date: Sun, 27 Oct 2002 08:47:28 -0500 (EST) Subject: [Python-Dev] "conflicting types" in python CVS build Message-ID: <200210271347.IAA25086@multics.mit.edu> > > hope this helps. > > Not really. Can you please copy line 169 of pgenmain.c? I can't see a > problem. at the time, lines 167-169 were char * PyOS_Readline(char *prompt) { which conflicted with the prototype in pythonrun.h. but it looks like a change which fixed that was checked in this morning. sorry for the confusion. alex. From skip@pobox.com Sun Oct 27 14:20:05 2002 From: skip@pobox.com (Skip Montanaro) Date: Sun, 27 Oct 2002 08:20:05 -0600 Subject: [Python-Dev] "conflicting types" in python CVS build In-Reply-To: <200210271347.IAA25086@multics.mit.edu> References: <200210271347.IAA25086@multics.mit.edu> Message-ID: <15803.62997.75987.884926@montanaro.dyndns.org> Alex> at the time, lines 167-169 were Alex> char * Alex> PyOS_Readline(char *prompt) Alex> { Alex> which conflicted with the prototype in pythonrun.h. but it looks Alex> like a change which fixed that was checked in this morning. sorry Alex> for the confusion. No confusion on your part. I encountered the same problem and checked in the change last night. This was the checkin message: Change PyOS_Readline declaration to match the recent change to myreadline.c (see Patch 512981). I changed stdin to sys_stdin in the body of the function, but did not change stderr to sys_stdout, though I suspect that may be the correct course. I don't know the code involved well enough to judge. Skip From guido@python.org Mon Oct 28 03:14:40 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 27 Oct 2002 22:14:40 -0500 Subject: [Python-Dev] __getitem__ in user defined classes In-Reply-To: Your message of "Mon, 21 Oct 2002 05:09:34 EDT." <20021021090934.GA99584@hishome.net> References: <002001c278ae$60425e00$5bf7a4d8@oemcomputer> <200210210339.g9L3drw15018@kuku.cosc.canterbury.ac.nz> <20021021090934.GA99584@hishome.net> Message-ID: <200210280314.g9S3EeR16822@pcp02138704pcs.reston01.va.comcast.net> > > > This could be simplified somewhat by making slices iterable > > > > Hmmm... if slices were iterable, they'd behave a lot > > like xrange objects. > > > > Perhaps > > > > slice == xrange > > > > could be true in some future Python? > > http://python.org/sf/575515 , rejected. Note that slice() isn't limited to integers -- that's up to whoever interprets the slice instance. But iteration really should only support ints. --Guido van Rossum (home page: http://www.python.org/~guido/) From python@rcn.com Mon Oct 28 13:43:59 2002 From: python@rcn.com (Raymond Hettinger) Date: Mon, 28 Oct 2002 08:43:59 -0500 Subject: [Python-Dev] Using defaultdict as globals/locals for eval() References: <20021025162651.GA9056@vader.boskant.nl> Message-ID: <003901c27e88$12271b40$d3b53bd0@oemcomputer> > I created a derived class of the standard `dict' that fills in a default > value when a key is not found. This is exactly the same as Guido describes > in his "descintro" paper. I tried to use this dictionary as the "globals" > parameter with eval(). As Guido already describes in his paper, this doesn't > work. I've wanted this for years and think it would be an important improvement. Unfortunately, it appears to be a problem without a simple solution. Raymond Hettinger From guido@python.org Mon Oct 28 14:14:53 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 28 Oct 2002 09:14:53 -0500 Subject: [Python-Dev] Using defaultdict as globals/locals for eval() In-Reply-To: Your message of "Mon, 28 Oct 2002 08:43:59 EST." <003901c27e88$12271b40$d3b53bd0@oemcomputer> References: <20021025162651.GA9056@vader.boskant.nl> <003901c27e88$12271b40$d3b53bd0@oemcomputer> Message-ID: <200210281414.g9SEErd18664@pcp02138704pcs.reston01.va.comcast.net> > > I created a derived class of the standard `dict' that fills in a > > default value when a key is not found. This is exactly the same as > > Guido describes in his "descintro" paper. I tried to use this > > dictionary as the "globals" parameter with eval(). As Guido > > already describes in his paper, this doesn't work. > > I've wanted this for years and think it would be an important > improvement. Unfortunately, it appears to be a problem without > a simple solution. The solution appears simple (use PyObject_GetItem etc. instead of PyDict_GetItem) but would cause serious performance hits: the dict API and its implementatio are highly optimized for this; PyObject_GetItem would add several levels of function calls, *plus* more reference count handling and exception handling. Oren Tirosh's global/builtin lookup speedup would be impossible. (Oren, how's that coming?) In addition, I'm concerned that this would become a popular tool to hack Python's semantics in all sorts of ways. While hacking the semantics is essential for some situations, in most cases this causes more problems than it solves because you can't use standard tools like PyChecker when you change something as fundamental as how variable references are resolved. --Guido van Rossum (home page: http://www.python.org/~guido/) From geertj@boskant.nl Mon Oct 28 14:44:43 2002 From: geertj@boskant.nl (Geert Jansen) Date: Mon, 28 Oct 2002 15:44:43 +0100 Subject: [Python-Dev] Using defaultdict as globals/locals for eval() In-Reply-To: <200210281414.g9SEErd18664@pcp02138704pcs.reston01.va.comcast.net> References: <20021025162651.GA9056@vader.boskant.nl> <003901c27e88$12271b40$d3b53bd0@oemcomputer> <200210281414.g9SEErd18664@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021028144443.GA21911@vader.boskant.nl> On Mon, Oct 28, 2002 at 09:14:53AM -0500, Guido van Rossum wrote: > > > I created a derived class of the standard `dict' that fills in a > > > default value when a key is not found. This is exactly the same as > > > Guido describes in his "descintro" paper. I tried to use this > > > dictionary as the "globals" parameter with eval(). As Guido > > > already describes in his paper, this doesn't work. > > > > I've wanted this for years and think it would be an important > > improvement. Unfortunately, it appears to be a problem without > > a simple solution. > > The solution appears simple (use PyObject_GetItem etc. instead of > PyDict_GetItem) but would cause serious performance hits: the dict API > and its implementatio are highly optimized for this; PyObject_GetItem > would add several levels of function calls, *plus* more reference > count handling and exception handling. Maybe eval() could check whether the "locals" and "globals" are plain dictionaries, or dictionary derived objects. This check would have to be done once, just after eval() is called. Depending on the result, PyDict_GetItem or PyObject_GetItem is used further on. Hmm, this sounds too trivial. I bet there are complications... Geert From mgilfix@eecs.tufts.edu Tue Oct 29 00:12:55 2002 From: mgilfix@eecs.tufts.edu (Michael Gilfix) Date: Mon, 28 Oct 2002 19:12:55 -0500 Subject: [Python-Dev] Valgrind on 2.2.2 In-Reply-To: <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> References: <20021017230108.GL9241@epoch.metaslash.com> <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021029001255.GB12789@eecs.tufts.edu> Just been lurkin' :) Possibly lost means that Valgrind didn't see any freeing for a specific point after a malloc. In valgrind terms, you probably forgot to free a pointer and lost all points that could reference that memory. In other words, a leak :) Usually 75% of those errors can be solved locally within the function, although I haven't looked at the specifics, but that 25% can get nasty. -- Mike On Thu, Oct 17 @ 20:38, Guido van Rossum wrote: > There are a few other records mentioning GC_Resize, but this one is > the biggest. Could it be that the free frame list is botched? OTOH, > what does "possibly lost" really mean? -- Michael Gilfix mgilfix@eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html From guido@python.org Tue Oct 29 00:37:55 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 28 Oct 2002 19:37:55 -0500 Subject: [Python-Dev] Valgrind on 2.2.2 In-Reply-To: Your message of "Mon, 28 Oct 2002 19:12:55 EST." <20021029001255.GB12789@eecs.tufts.edu> References: <20021017230108.GL9241@epoch.metaslash.com> <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> <20021029001255.GB12789@eecs.tufts.edu> Message-ID: <200210290037.g9T0bt822570@pcp02138704pcs.reston01.va.comcast.net> > Just been lurkin' :) Possibly lost means that Valgrind didn't see > any freeing for a specific point after a malloc. In valgrind terms, > you probably forgot to free a pointer and lost all points that could > reference that memory. In other words, a leak :) Usually 75% of those > errors can be solved locally within the function, although I haven't > looked at the specifics, but that 25% can get nasty. Booh. That statistic doesn't apply to Python -- there are no "simple" leaks left in Python's C code, the remaining ones are all nasty. Also, valgrind carefully distinguishes between "definitely lost" and "possibly lost". What's the difference? --Guido van Rossum (home page: http://www.python.org/~guido/) From mgilfix@eecs.tufts.edu Tue Oct 29 01:00:29 2002 From: mgilfix@eecs.tufts.edu (Michael Gilfix) Date: Mon, 28 Oct 2002 20:00:29 -0500 Subject: [Python-Dev] Valgrind on 2.2.2 In-Reply-To: <200210290037.g9T0bt822570@pcp02138704pcs.reston01.va.comcast.net> References: <20021017230108.GL9241@epoch.metaslash.com> <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> <20021029001255.GB12789@eecs.tufts.edu> <200210290037.g9T0bt822570@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021029010029.GC12789@eecs.tufts.edu> This should help shed some light on the situation: Quoth the docs: """ For each such block, Valgrind scans the entire address space of the process, looking for pointers to the block. One of three situations may result: * A pointer to the start of the block is found. This usually indicates programming sloppiness; since the block is still pointed at, the programmer could, at least in principle, free'd it before program exit. * A pointer to the interior of the block is found. The pointer might originally have pointed to the start and have been moved along, or it might be entirely unrelated. Valgrind deems such a block as "dubious", that is, possibly leaked, because it's unclear whether or not a pointer to it still exists. * The worst outcome is that no pointer to the block can be found. The block is classified as "leaked", because the programmer could not possibly have free'd it at program exit, since no pointer to it exists. This might be a symptom of having lost the pointer at some earlier point in the program. """ Possibly is the second case and definitely lost is the third case. The definitely lost, in my experience, tends to mean you just forgot to free a pointer. The possibly lost usually means that some memory rot occurred, where it's not clear which pointer is causing the mem leak. -- Mike On Mon, Oct 28 @ 19:37, Guido van Rossum wrote: > > Just been lurkin' :) Possibly lost means that Valgrind didn't see > > any freeing for a specific point after a malloc. In valgrind terms, > > you probably forgot to free a pointer and lost all points that could > > reference that memory. In other words, a leak :) Usually 75% of those > > errors can be solved locally within the function, although I haven't > > looked at the specifics, but that 25% can get nasty. > > Booh. > > That statistic doesn't apply to Python -- there are no "simple" leaks > left in Python's C code, the remaining ones are all nasty. > > Also, valgrind carefully distinguishes between "definitely lost" and > "possibly lost". What's the difference? > > --Guido van Rossum (home page: http://www.python.org/~guido/) -- Michael Gilfix mgilfix@eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html From guido@python.org Tue Oct 29 01:15:44 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 28 Oct 2002 20:15:44 -0500 Subject: [Python-Dev] Valgrind on 2.2.2 In-Reply-To: Your message of "Mon, 28 Oct 2002 20:00:29 EST." <20021029010029.GC12789@eecs.tufts.edu> References: <20021017230108.GL9241@epoch.metaslash.com> <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> <20021029001255.GB12789@eecs.tufts.edu> <200210290037.g9T0bt822570@pcp02138704pcs.reston01.va.comcast.net> <20021029010029.GC12789@eecs.tufts.edu> Message-ID: <200210290115.g9T1Fih22665@pcp02138704pcs.reston01.va.comcast.net> > This should help shed some light on the situation: Thanks; this indeed helps. > Quoth the docs: > > """ > For each such block, Valgrind scans the entire address space of the > process, looking for pointers to the block. One of three situations > may result: > > * A pointer to the start of the block is found. This usually > indicates programming sloppiness; since the block is still pointed > at, the programmer could, at least in principle, free'd it before > program exit. Booh again. Lots of globals get initialized with pointers to malloc'ed blocks that are never freed. There are never called "leaks" in other leak detectors, just "alive at exit". I think valgrind actually doesn't call these leaks either. > * A pointer to the interior of the block is found. The pointer > might originally have pointed to the start and have been moved > along, or it might be entirely unrelated. Valgrind deems such a > block as "dubious", that is, possibly leaked, because it's unclear > whether or not a pointer to it still exists. Aha! This may be the case. When an object has a GC header, all pointers to the object point to an address 12 bytes in the block, which is where the "object" lay-out begins. Normally, there should be at least one pointer to the start of the block from one of the GC chains, but objects don't have to be in a chain at all. (I wonder if pymalloc adds to the confusion, since its arenas count as a single block to malloc and hence to valgrind, but are internally cut up into many objects.) > * The worst outcome is that no pointer to the block can be > found. The block is classified as "leaked", because the programmer > could not possibly have free'd it at program exit, since no pointer > to it exists. This might be a symptom of having lost the pointer at > some earlier point in the program. This is a true leak. > """ > > Possibly is the second case and definitely lost is the third case. > The definitely lost, in my experience, tends to mean you just forgot > to free a pointer. The possibly lost usually means that some memory > rot occurred, where it's not clear which pointer is causing the mem > leak. How much Python extension coding (in C) have you done? In Python, it almost never is a matter of forgetting to free() -- it's usually a matter of forgetting to DECREF, and sometimes a matter of doing an unnecessary INCREF. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Tue Oct 29 01:36:50 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 28 Oct 2002 20:36:50 -0500 Subject: [Python-Dev] Valgrind on 2.2.2 In-Reply-To: <200210290115.g9T1Fih22665@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido] > ... > (I wonder if pymalloc adds to the confusion, since its arenas count as > a single block to malloc and hence to valgrind, but are internally cut > up into many objects.) For each arena, the address returned by malloc() is stored in the file-static arenas[] vector. So if a diagnostic program can find that vector, it can find the base address of every arena gotten from malloc. In a debug build, though, pymalloc adds pad bytes to both ends of each request (and whether handled by pymalloc or by malloc!), and returns the address of the byte beyond the leading pad byte. This can leave any number of system-malloc blocks with no direct pointer to their start. That's specific to the debug build, which forces all Python mem API calls to go thru pymalloc (the release build only directs PyObject_{Malloc,etc}() calls to pymalloc). From mgilfix@eecs.tufts.edu Tue Oct 29 01:58:45 2002 From: mgilfix@eecs.tufts.edu (Michael Gilfix) Date: Mon, 28 Oct 2002 20:58:45 -0500 Subject: [Python-Dev] Valgrind on 2.2.2 In-Reply-To: <200210290115.g9T1Fih22665@pcp02138704pcs.reston01.va.comcast.net> References: <20021017230108.GL9241@epoch.metaslash.com> <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> <20021029001255.GB12789@eecs.tufts.edu> <200210290037.g9T0bt822570@pcp02138704pcs.reston01.va.comcast.net> <20021029010029.GC12789@eecs.tufts.edu> <200210290115.g9T1Fih22665@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021029015845.GD12789@eecs.tufts.edu> On Mon, Oct 28 @ 20:15, Guido van Rossum wrote: > > * A pointer to the start of the block is found. This usually > > indicates programming sloppiness; since the block is still pointed > > at, the programmer could, at least in principle, free'd it before > > program exit. > > Booh again. Lots of globals get initialized with pointers to > malloc'ed blocks that are never freed. There are never called "leaks" > in other leak detectors, just "alive at exit". I think valgrind > actually doesn't call these leaks either. I don't think valgrind reports this first case. Only the second and third, from what I gathered in the docs. > > Possibly is the second case and definitely lost is the third case. > > The definitely lost, in my experience, tends to mean you just forgot > > to free a pointer. The possibly lost usually means that some memory > > rot occurred, where it's not clear which pointer is causing the mem > > leak. > > How much Python extension coding (in C) have you done? In Python, it > almost never is a matter of forgetting to free() -- it's usually a > matter of forgetting to DECREF, and sometimes a matter of doing an > unnecessary INCREF. I've done a fair bit, but to explain valgrind things, I figured it was best to talk the valgrind talk. Either way, it comes to the same thing. Glad that helped. -- Mike -- Michael Gilfix mgilfix@eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html From jepler@unpythonic.net Tue Oct 29 02:39:55 2002 From: jepler@unpythonic.net (Jeff Epler) Date: Mon, 28 Oct 2002 20:39:55 -0600 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec Message-ID: <20021028203952.A1460@unpythonic.net> --PEIAKu/WMn1b1Hv9 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Using dict subclasses in eval() and exec ---------------------------------------- With the attached patch, you can now use a dict subclass which implements __getitem__ as the global or builtin namespace for eval() and exec. If you do not use the feature, the performance impact is low (<2%). The fast case adds only one test+branch to each eval_code() call (to set 'fastglobals' and 'slowbuiltins'), each lookup in globals (to check 'fastglobals'), and each lookup in builtins (to check 'slowbuiltins'). If you do use the feature, the performance impact is unfortunately quite substantial (400%). If you use subclasses of dict with eval()/exec, but do not define __getitem__, the performance penalty is modest (~10%). The smaller penalty of using a dict subclass without __getitem__ could probably be erased if the 'fastglobals' test can check that __getitem__ is not defined Included are a patch and a test suite. Note that the patch causes a failure in test_descr which is testing that __getitem__ is *not* called on a dictionary subclass. Pystone timings on a 1GHz Duron system running RedHat Linux 8.0 (gcc 3.2) follow. CVS +patch +subclass +__getattr__ 1 2.74 2.76 3 14.05 2 2.74 2.78 3.02 14.07 3 2.74 2.78 3.03 14.07 4 2.76 2.78 3.04 14.08 5 2.76 2.79 3.04 14.1 avg 2.748 2.778 3.026 14.074 % 100 101.1 110.1 512.2 * 2.3a0 CVS (28 Oct 2002) Pystone(1.1) time for 50000 passes = 2.74 This machine benchmarks at 18248.2 pystones/second Pystone(1.1) time for 50000 passes = 2.74 This machine benchmarks at 18248.2 pystones/second Pystone(1.1) time for 50000 passes = 2.74 This machine benchmarks at 18248.2 pystones/second Pystone(1.1) time for 50000 passes = 2.76 This machine benchmarks at 18115.9 pystones/second Pystone(1.1) time for 50000 passes = 2.76 This machine benchmarks at 18115.9 pystones/second * 2.3a0 CVS + patch Pystone(1.1) time for 50000 passes = 2.78 This machine benchmarks at 17985.6 pystones/second Pystone(1.1) time for 50000 passes = 2.78 This machine benchmarks at 17985.6 pystones/second Pystone(1.1) time for 50000 passes = 2.79 This machine benchmarks at 17921.1 pystones/second Pystone(1.1) time for 50000 passes = 2.79 This machine benchmarks at 17921.1 pystones/second Pystone(1.1) time for 50000 passes = 2.76 This machine benchmarks at 18115.9 pystones/second * 2.3a0 CVS + patch + dict subclass w/o __getitem__ (commandline: ./python -c 'class D(dict): pass d = D(globals()); execfile("Lib/test/pystone.py", d)') Pystone(1.1) time for 50000 passes = 3.03 This machine benchmarks at 16501.7 pystones/second Pystone(1.1) time for 50000 passes = 3.04 This machine benchmarks at 16447.4 pystones/second Pystone(1.1) time for 50000 passes = 3 This machine benchmarks at 16666.7 pystones/second Pystone(1.1) time for 50000 passes = 3.04 This machine benchmarks at 16447.4 pystones/second Pystone(1.1) time for 50000 passes = 3.02 This machine benchmarks at 16556.3 pystones/second * 2.3a0 CVS + patch + dict subclass w/__getitem__ (commandline: ./python -c 'class D(dict): __getitem__ = lambda s, i: dict.__getitem__(s, i) d = D(globals()); execfile("Lib/test/pystone.py", d)') Pystone(1.1) time for 50000 passes = 14.05 This machine benchmarks at 3558.72 pystones/second Pystone(1.1) time for 50000 passes = 14.08 This machine benchmarks at 3551.14 pystones/second Pystone(1.1) time for 50000 passes = 14.07 This machine benchmarks at 3553.66 pystones/second Pystone(1.1) time for 50000 passes = 14.07 This machine benchmarks at 3553.66 pystones/second Pystone(1.1) time for 50000 passes = 14.1 This machine benchmarks at 3546.1 pystones/second --PEIAKu/WMn1b1Hv9 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="ceval-userdict.patch" Index: Python/ceval.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/ceval.c,v retrieving revision 2.338 diff -u -r2.338 ceval.c --- Python/ceval.c 3 Oct 2002 09:53:11 -0000 2.338 +++ Python/ceval.c 29 Oct 2002 00:22:42 -0000 @@ -529,6 +529,8 @@ char *filename; #endif + int fastglobals, slowbuiltins; + /* Tuple access macros */ #ifndef Py_DEBUG @@ -603,6 +605,9 @@ names = co->co_names; consts = co->co_consts; fastlocals = f->f_localsplus; +#define PyDict_CheckExact(d) ((d)->ob_type == &PyDict_Type) + fastglobals = PyDict_CheckExact(f->f_globals); + slowbuiltins = !PyDict_CheckExact(f->f_builtins); freevars = f->f_localsplus + f->f_nlocals; _PyCode_GETCODEPTR(co, &first_instr); /* An explanation is in order for the next line. @@ -1596,7 +1601,7 @@ PyObject_REPR(w)); break; } - err = PyDict_SetItem(x, w, v); + err = (fastglobals ? PyDict_SetItem : PyObject_SetItem)(x, w, v); Py_DECREF(v); break; @@ -1675,7 +1680,7 @@ case STORE_GLOBAL: w = GETITEM(names, oparg); v = POP(); - err = PyDict_SetItem(f->f_globals, w, v); + err = (fastglobals ? PyDict_SetItem : PyObject_SetItem)(f->f_globals, w, v); Py_DECREF(v); break; @@ -1696,9 +1701,10 @@ } x = PyDict_GetItem(x, w); if (x == NULL) { - x = PyDict_GetItem(f->f_globals, w); + x = (fastglobals ? PyDict_GetItem : PyObject_GetItem)(f->f_globals, w); if (x == NULL) { - x = PyDict_GetItem(f->f_builtins, w); + if (!fastglobals) PyErr_Clear(); + x = (slowbuiltins ? PyObject_GetItem : PyDict_GetItem)(f->f_builtins, w); if (x == NULL) { format_exc_check_arg( PyExc_NameError, @@ -1711,9 +1717,10 @@ PUSH(x); break; + case LOAD_GLOBAL: w = GETITEM(names, oparg); - if (PyString_CheckExact(w)) { + if (PyString_CheckExact(w) && fastglobals) { /* Inline the PyDict_GetItem() calls. WARNING: this is an extreme speed hack. Do not try this at home. */ @@ -1727,6 +1734,7 @@ PUSH(x); continue; } + if(slowbuiltins) goto load_builtins_slow; d = (PyDictObject *)(f->f_builtins); x = d->ma_lookup(d, w, hash)->me_value; if (x != NULL) { @@ -1738,9 +1746,11 @@ } } /* This is the un-inlined version of the code above */ - x = PyDict_GetItem(f->f_globals, w); + x = PyObject_GetItem(f->f_globals, w); if (x == NULL) { - x = PyDict_GetItem(f->f_builtins, w); + PyErr_Clear(); + load_builtins_slow: + x = PyObject_GetItem(f->f_builtins, w); if (x == NULL) { load_global_error: format_exc_check_arg( --PEIAKu/WMn1b1Hv9 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="test_dictsubclass.py" X='xx' class D(dict): def __getitem__(self, item): if item is X: return X return dict.__getitem__(self, item) d = D(globals()) d['zz'] = 'zz' # Make sure the dict acts like it should print d[X] == X print d['D'] == D print d['zz'] == 'zz' try: d['yy'] except KeyError, detail: print "KeyError", detail else: print "failed to get exception" # Make sure that exec and eval() in globals() works right exec "print (len.__name__, D.__name__)" in globals() exec "print (lambda: (len.__name__, D.__name__))()" in globals() print eval("len.__name__, D.__name__", globals()) print eval("(lambda: (len.__name__, D.__name__))()", globals()) try: eval(X, globals()) except NameError, detail: print detail else: print "failed to get exception" try: exec "zz" in globals() except NameError, detail: print detail else: print "failed to get exception" try: exec "yy" in globals() except NameError, detail: print detail else: print "failed to get exception" # Make sure that exec and eval() in d works right exec "print (len.__name__, D.__name__, xx, zz)" in d exec "print (lambda: (len.__name__, D.__name__, xx, zz))()" in d print eval("len.__name__, D.__name__, xx, zz", d) print eval("(lambda: (len.__name__, D.__name__, xx, zz))()", d) try: exec "yy" in d except NameError, detail: print detail else: print "failed to get exception" --PEIAKu/WMn1b1Hv9 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=test_dictsubclass test_dictsubclass True True True KeyError 'yy' ('len', 'D') ('len', 'D') ('len', 'D') ('len', 'D') name 'xx' is not defined name 'zz' is not defined name 'yy' is not defined ('len', 'D', 'xx', 'zz') ('len', 'D', 'xx', 'zz') ('len', 'D', 'xx', 'zz') ('len', 'D', 'xx', 'zz') name 'yy' is not defined --PEIAKu/WMn1b1Hv9-- From jepler@unpythonic.net Tue Oct 29 02:42:35 2002 From: jepler@unpythonic.net (Jeff Epler) Date: Mon, 28 Oct 2002 20:42:35 -0600 Subject: [Python-Dev] Valgrind on 2.2.2 In-Reply-To: <20021029015845.GD12789@eecs.tufts.edu> References: <20021017230108.GL9241@epoch.metaslash.com> <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> <20021029001255.GB12789@eecs.tufts.edu> <200210290037.g9T0bt822570@pcp02138704pcs.reston01.va.comcast.net> <20021029010029.GC12789@eecs.tufts.edu> <200210290115.g9T1Fih22665@pcp02138704pcs.reston01.va.comcast.net> <20021029015845.GD12789@eecs.tufts.edu> Message-ID: <20021028204234.A1528@unpythonic.net> On Mon, Oct 28, 2002 at 08:58:45PM -0500, Michael Gilfix wrote: > I don't think valgrind reports this first case [a pointer to the > beginning of the block is found]. Only the second and > third, from what I gathered in the docs. See --show-reachable From neal@metaslash.com Tue Oct 29 03:40:16 2002 From: neal@metaslash.com (Neal Norwitz) Date: Mon, 28 Oct 2002 22:40:16 -0500 Subject: [Python-Dev] Valgrind on 2.2.2 In-Reply-To: <20021028204234.A1528@unpythonic.net> References: <20021017230108.GL9241@epoch.metaslash.com> <200210180038.g9I0cef12864@pcp02138704pcs.reston01.va.comcast.net> <20021029001255.GB12789@eecs.tufts.edu> <200210290037.g9T0bt822570@pcp02138704pcs.reston01.va.comcast.net> <20021029010029.GC12789@eecs.tufts.edu> <200210290115.g9T1Fih22665@pcp02138704pcs.reston01.va.comcast.net> <20021029015845.GD12789@eecs.tufts.edu> <20021028204234.A1528@unpythonic.net> Message-ID: <20021029034016.GE17007@epoch.metaslash.com> On Mon, Oct 28, 2002 at 08:42:35PM -0600, Jeff Epler wrote: > On Mon, Oct 28, 2002 at 08:58:45PM -0500, Michael Gilfix wrote: > > I don't think valgrind reports this first case [a pointer to the > > beginning of the block is found]. Only the second and > > third, from what I gathered in the docs. > > See --show-reachable --show-reachable was used when running valgrind. Python was not built using pydebug. Neal From mal@lemburg.com Tue Oct 29 07:57:01 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 29 Oct 2002 08:57:01 +0100 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec References: <20021028203952.A1460@unpythonic.net> Message-ID: <3DBE3F4D.1000301@lemburg.com> Jeff Epler wrote: > Using dict subclasses in eval() and exec > ---------------------------------------- > > With the attached patch, you can now use a dict subclass which implements > __getitem__ as the global or builtin namespace for eval() and exec. > > If you do not use the feature, the performance impact is low (<2%). > The fast case adds only one test+branch to each eval_code() call (to > set 'fastglobals' and 'slowbuiltins'), each lookup in globals (to > check 'fastglobals'), and each lookup in builtins (to check 'slowbuiltins'). > > If you do use the feature, the performance impact is unfortunately > quite substantial (400%). > > If you use subclasses of dict with eval()/exec, but do not define __getitem__, > the performance penalty is modest (~10%). If all you want is a way to define a default value for a failing dictionary lookup, wouldn't it be better to simply add this feature to the standard dictionary implementation ? I don't think that many people will use this feature, yet everybody would pay for the performance hit. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mhammond@skippinet.com.au Tue Oct 29 09:54:45 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Tue, 29 Oct 2002 20:54:45 +1100 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: <3DBE3F4D.1000301@lemburg.com> Message-ID: > I don't think that many people will use this feature, yet everybody > would pay for the performance hit. I wonder how many reasonable real-life scenarios actually exist for this feature? My personal use would be for COM, to implement case-insensitive namespaces. In some cases, case-insensitivity is actually part of the implementation specification, and I can imagine other scenarios where case insensitive namespaces are "reasonable" even in the face of Python's excellent decision not to go this route. I wonder is this is actually the root of many requests to see this feature implemented? If not, it would be interesting to see what the competing desires really are. Mark. From guido@python.org Tue Oct 29 12:09:22 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 29 Oct 2002 07:09:22 -0500 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: Your message of "Mon, 28 Oct 2002 20:39:55 CST." <20021028203952.A1460@unpythonic.net> References: <20021028203952.A1460@unpythonic.net> Message-ID: <200210291209.g9TC9M924037@pcp02138704pcs.reston01.va.comcast.net> > @@ -1696,9 +1701,10 @@ > } > x = PyDict_GetItem(x, w); > if (x == NULL) { > - x = PyDict_GetItem(f->f_globals, w); > + x = (fastglobals ? PyDict_GetItem : PyObject_GetItem)(f->f_globals, w); > if (x == NULL) { > - x = PyDict_GetItem(f->f_builtins, w); > + if (!fastglobals) PyErr_Clear(); > + x = (slowbuiltins ? PyObject_GetItem : PyDict_GetItem)(f->f_builtins, w); > if (x == NULL) { > format_exc_check_arg( > PyExc_NameError, I don't see the extra DECREF here (and below) needed to compensate for the fact that PyObject_GetItem() returns the object with incremented refcount but PyDict_GetItem() doesn't. What are you going to do with all the *other* places where PyDict_GetItem() is used? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Oct 29 12:43:50 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 29 Oct 2002 07:43:50 -0500 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: Your message of "Tue, 29 Oct 2002 20:54:45 +1100." References: Message-ID: <200210291243.g9TChoK24426@pcp02138704pcs.reston01.va.comcast.net> > I wonder how many reasonable real-life scenarios actually exist for > this feature? > > My personal use would be for COM, to implement case-insensitive > namespaces. In some cases, case-insensitivity is actually part of > the implementation specification, and I can imagine other scenarios > where case insensitive namespaces are "reasonable" even in the face > of Python's excellent decision not to go this route. > > I wonder is this is actually the root of many requests to see this > feature implemented? If not, it would be interesting to see what > the competing desires really are. Excellent observation. Perhaps the people who want this can get together and write a PEP that explains the motivation and why the alternatives don't work? (E.g. when you need case-insensitive variable names, perhaps it's just as easy to preprocess the source code?) --Guido van Rossum (home page: http://www.python.org/~guido/) From jepler@unpythonic.net Tue Oct 29 14:34:26 2002 From: jepler@unpythonic.net (Jeff Epler) Date: Tue, 29 Oct 2002 08:34:26 -0600 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: <200210291209.g9TC9M924037@pcp02138704pcs.reston01.va.comcast.net> References: <20021028203952.A1460@unpythonic.net> <200210291209.g9TC9M924037@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021029143416.GA16099@unpythonic.net> On Tue, Oct 29, 2002 at 07:09:22AM -0500, Guido van Rossum wrote: > I don't see the extra DECREF here (and below) needed to compensate for > the fact that PyObject_GetItem() returns the object with incremented > refcount but PyDict_GetItem() doesn't. oops. I believe this can be fixed, if only by a DECREF in all paths and an INCREF in the fast path. However, this will worsen the "fast case" performance figures at least a tiny bit. [guido again] > What are you going to do with all the *other* places where > PyDict_GetItem() is used? Which places do you have in mind? The common feature request seems to be for globals() only, and I can't immediately think of a way of accessing globals() that my patch doesn't (attempt to) cover. Unfortunately, I'm not particularly excited about this capability myself. What I think this does show, however, is that using a sequence like (not quite C code) inline PyObject *PyObject_GetItem_FastForDicts(o, k) { if(PyDict_CheckExact(o)) { ret = PyDict_GetItem(o, k); Py_INCREF(ret); return ret; } return PyObject_GetItem(o, k); } will impose in-the-noise penalties for 'dict' instances, modest penalties for 'dict' subclass instances without __getitem__, and provide the full power for general objects. On Tue, Oct 29, 2002 at 08:57:01AM +0100, M.-A. Lemburg wrote: > If all you want is a way to define a default value for a failing > dictionary lookup, wouldn't it be better to simply add this feature > to the standard dictionary implementation ? It doesn't seem to be quite this simple, because you almost certainly want to have items in the builtin namespace appear even if you want to give a default on what would eventually be a NameError for a LOAD_GLOBAL/LOAD_NAME. You end up with something like class D(dict): def __getitem__(self, item): try: return dict.__getitem__(self, item) except KeyError: try: return getattr(__builtins__, item) except AttributeError: return "XX" which is more complicated that a "simple" defaulting dictionary. Unfortunately, this is another time when I am not a strong proponent of a feature, but merely wrote the patch to see if the "common wisdom" on which the rejection was based is valid. (common wisdom's been right in earlier cases, but I'm tempted to call the speed penalties for this change entirely tolerable if the purposes of people who do want this patch are good ones) On Tue, Oct 29, 2002 at 12:11:12AM -0500, Raymond Hettinger wrote [in a private message]: > Do you know why the following snippet doesn't work with your patch? > > class SpreadSheet(dict): > def __getitem__(self, key): > return eval(dict.__getitem__(self,key), self) > > ss = SpreadSheet() > ss['a1'] = '5' > ss['a2'] = 'a1*6' > ss['a3'] = 'a2*a1' > print ss['a3'] It took me awhile to figure this out... LOAD_NAME first tries to load from the locals dict, where I did not change the way that name resolution functions. eval() uses the same dict for locals and globals if the locals (third arg) are not directly specified. The following works: class SpreadSheet(dict): def __getitem__(self, key): return eval(dict.__getitem__(self,key), self, {}) Jeff From guido@python.org Tue Oct 29 15:53:16 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 29 Oct 2002 10:53:16 -0500 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: Your message of "Tue, 29 Oct 2002 08:34:26 CST." <20021029143416.GA16099@unpythonic.net> References: <20021028203952.A1460@unpythonic.net> <200210291209.g9TC9M924037@pcp02138704pcs.reston01.va.comcast.net> <20021029143416.GA16099@unpythonic.net> Message-ID: <200210291553.g9TFrGJ01406@odiug.zope.com> > > I don't see the extra DECREF here (and below) needed to compensate for > > the fact that PyObject_GetItem() returns the object with incremented > > refcount but PyDict_GetItem() doesn't. > > oops. I believe this can be fixed, if only by a DECREF in all paths > and an INCREF in the fast path. However, this will worsen the "fast > case" performance figures at least a tiny bit. I'm already unhappy about the performance hit of your patch. We're trying very hard to make global and built-in access as fast as possible. This patch adds extra local variable references to the "fast path". More INCREFS and DECREFS will slow it down even more. Not just because there's more executed code, but also because there's more code, period, which may reduce the I-cache effectiveness. > [guido again] > > What are you going to do with all the *other* places where > > PyDict_GetItem() is used? > > Which places do you have in mind? The common feature request seems > to be for globals() only, and I can't immediately think of a way of > accessing globals() that my patch doesn't (attempt to) cover. There are lots of other places where PyDict_*Item is used for an object that could be a dict subclass. Unless you've tracked all these down I think it wouldn't be wise to apply this patch. We either say (as we say now) "you can't override __getitem__ in a dict passed into the Python interpreter and expect it to work" or we say "you can override __getitem__ and it will work". It would be really bad to say "you can override __getitem__ and it will usually work but not in these twelve situations, because we didn't bother to change legacy code." > Unfortunately, I'm not particularly excited about this capability > myself. Me neither, to say the least. I still feel hostile, both because of the performance hit, and because I worry that it will be used to create obfuscated code that *looks* like Python but isn't Python. > What I think this does show, however, is that using a sequence like > (not quite C code) > inline PyObject *PyObject_GetItem_FastForDicts(o, k) { > if(PyDict_CheckExact(o)) { > ret = PyDict_GetItem(o, k); > Py_INCREF(ret); > return ret; > } > return PyObject_GetItem(o, k); > } > will impose in-the-noise penalties for 'dict' instances, modest > penalties for 'dict' subclass instances without __getitem__, and > provide the full power for general objects. I'd like to see more benchmarking of this (as well as more motivation). > On Tue, Oct 29, 2002 at 08:57:01AM +0100, M.-A. Lemburg wrote: > > If all you want is a way to define a default value for a failing > > dictionary lookup, wouldn't it be better to simply add this feature > > to the standard dictionary implementation ? > > It doesn't seem to be quite this simple, because you almost certainly > want to have items in the builtin namespace appear even if you want to > give a default on what would eventually be a NameError for a > LOAD_GLOBAL/LOAD_NAME. You end up with something like > class D(dict): > def __getitem__(self, item): > try: > return dict.__getitem__(self, item) > except KeyError: > try: > return getattr(__builtins__, item) > except AttributeError: > return "XX" > which is more complicated that a "simple" defaulting dictionary. > > Unfortunately, this is another time when I am not a strong proponent of a > feature, but merely wrote the patch to see if the "common wisdom" on which > the rejection was based is valid. (common wisdom's been right in earlier > cases, but I'm tempted to call the speed penalties for this change > entirely tolerable if the purposes of people who do want this patch are > good ones) > > > On Tue, Oct 29, 2002 at 12:11:12AM -0500, Raymond Hettinger wrote > [in a private message]: > > Do you know why the following snippet doesn't work with your patch? > > > > class SpreadSheet(dict): > > def __getitem__(self, key): > > return eval(dict.__getitem__(self,key), self) > > > > ss = SpreadSheet() > > ss['a1'] = '5' > > ss['a2'] = 'a1*6' > > ss['a3'] = 'a2*a1' > > print ss['a3'] > > It took me awhile to figure this out... > > LOAD_NAME first tries to load from the locals dict, where I did not > change the way that name resolution functions. Another deficiency of the patch. > eval() uses > the same dict for locals and globals if the locals (third arg) are not > directly specified. > > The following works: > class SpreadSheet(dict): > def __getitem__(self, key): > return eval(dict.__getitem__(self,key), self, {}) > > Jeff --Guido van Rossum (home page: http://www.python.org/~guido/) From jepler@unpythonic.net Tue Oct 29 17:10:25 2002 From: jepler@unpythonic.net (Jeff Epler) Date: Tue, 29 Oct 2002 11:10:25 -0600 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: <200210291553.g9TFrGJ01406@odiug.zope.com> References: <20021028203952.A1460@unpythonic.net> <200210291209.g9TC9M924037@pcp02138704pcs.reston01.va.comcast.net> <20021029143416.GA16099@unpythonic.net> <200210291553.g9TFrGJ01406@odiug.zope.com> Message-ID: <20021029171024.GC18602@unpythonic.net> Still working from the version which is missing DECREFs, I get these timings from a program intended to more directly measure the speed of local, global, and builtin name lookups: Vanilla Python CVS. In effect these are three runs of the same thing, so you get some idea of the timing variations on my system. $ python-cvs/python lookup-speed.py Timing 4000000 name lookups type builtin global local overhead dict 2.74015 1.99215 1.21323 0.17610 D 2.69901 1.99129 1.24662 0.17832 E 2.73839 2.03171 1.25233 0.17843 CVS with my original patch (leaks references). "D" is a dict subclass which doesn't define __getitem__, "E" is a dict subclass which does (simply calling dict.__getitem__ and returning the result) $ python-dict/python lookup-speed.py Timing 4000000 name lookups type builtin global local overhead dict 2.81268 2.04795 1.26585 0.18117 D 5.08308 2.85853 1.23625 0.17944 E 81.62213 64.60481 1.26524 0.17973 This is a ~3% slowdown on the test with the heaviest use of builtins and globals that I can think of offhand. Slowdown: builtin global dict 3% 3% D 80% 40% E *28 *31 I'm about done pretending to cheerlead for this patch, though if I fix the reference problem I'll post the revised diffs. Jeff From jeremy@alum.mit.edu Tue Oct 29 17:14:53 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Tue, 29 Oct 2002 12:14:53 -0500 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: <20021029171024.GC18602@unpythonic.net> References: <20021028203952.A1460@unpythonic.net> <200210291209.g9TC9M924037@pcp02138704pcs.reston01.va.comcast.net> <20021029143416.GA16099@unpythonic.net> <200210291553.g9TFrGJ01406@odiug.zope.com> <20021029171024.GC18602@unpythonic.net> Message-ID: <15806.49677.886970.115803@slothrop.zope.com> >>>>> "JE" == Jeff Epler writes: JE> I'm about done pretending to cheerlead for this patch, though if JE> I fix the reference problem I'll post the revised diffs. I'm not enthusiastic about this patch, but I think it's really helpful that you've done this work. A concrete patch, albeit incomplete, and some real performance measurements improves the quality of discussion on this issue. Did you post the lookup-speed.py script in an earlier message? I don't think I know what exactly you are measuring. Jeremy From guido@python.org Tue Oct 29 17:14:57 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 29 Oct 2002 12:14:57 -0500 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: Your message of "Tue, 29 Oct 2002 11:10:25 CST." <20021029171024.GC18602@unpythonic.net> References: <20021028203952.A1460@unpythonic.net> <200210291209.g9TC9M924037@pcp02138704pcs.reston01.va.comcast.net> <20021029143416.GA16099@unpythonic.net> <200210291553.g9TFrGJ01406@odiug.zope.com> <20021029171024.GC18602@unpythonic.net> Message-ID: <200210291714.g9THEwP02615@odiug.zope.com> 1. You've got to show your test code. 2. Have you tried pybench? --Guido van Rossum (home page: http://www.python.org/~guido/) From jepler@unpythonic.net Tue Oct 29 17:18:29 2002 From: jepler@unpythonic.net (Jeff Epler) Date: Tue, 29 Oct 2002 11:18:29 -0600 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: <20021029171024.GC18602@unpythonic.net> References: <20021028203952.A1460@unpythonic.net> <200210291209.g9TC9M924037@pcp02138704pcs.reston01.va.comcast.net> <20021029143416.GA16099@unpythonic.net> <200210291553.g9TFrGJ01406@odiug.zope.com> <20021029171024.GC18602@unpythonic.net> Message-ID: <20021029171828.GD18602@unpythonic.net> Here's the lookup-speed script. num = 1000000 code = """ x = None r = range(%d) def builtin_lookup(): for i in r: len; len; len; len len; len; len; len len; len; len; len len; len; len; len def global_lookup(): for i in r: x; x; x; x x; x; x; x x; x; x; x x; x; x; x def local_lookup(): x = None for i in r: x; x; x; x x; x; x; x x; x; x; x x; x; x; x def overhead(): for i in r: pass """ % num class D(dict): pass class E(dict): __getitem__ = lambda s, k: dict.__getitem__(s, k) print "Timing %d name lookups" % (num*4) print "%8s %8s %8s %8s %8s" % ("type", "builtin", "global", "local", "overhead") import time for dc in dict, D, E: d = dc() exec code in d print "%8s" % dc.__name__, for f in ("builtin_lookup()", "global_lookup()", "local_lookup()", "overhead()"): t0 = time.time() exec f in d t1 = time.time() print "%8.5f" % (t1-t0), print From guido@python.org Tue Oct 29 17:28:29 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 29 Oct 2002 12:28:29 -0500 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: Your message of "Tue, 29 Oct 2002 12:14:53 EST." <15806.49677.886970.115803@slothrop.zope.com> References: <20021028203952.A1460@unpythonic.net> <200210291209.g9TC9M924037@pcp02138704pcs.reston01.va.comcast.net> <20021029143416.GA16099@unpythonic.net> <200210291553.g9TFrGJ01406@odiug.zope.com> <20021029171024.GC18602@unpythonic.net> <15806.49677.886970.115803@slothrop.zope.com> Message-ID: <200210291728.g9THSTb06525@odiug.zope.com> BTW, the (final) patch, test script, and commentary should be uploaded to the SF patch manager. --Guido van Rossum (home page: http://www.python.org/~guido/) From DavidA@ActiveState.com Tue Oct 29 18:20:28 2002 From: DavidA@ActiveState.com (David Ascher) Date: Tue, 29 Oct 2002 10:20:28 -0800 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec References: <200210291243.g9TChoK24426@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3DBED16C.90801@ActiveState.com> Guido van Rossum wrote: >>I wonder how many reasonable real-life scenarios actually exist for >>this feature? >> >>My personal use would be for COM, to implement case-insensitive >>namespaces. In some cases, case-insensitivity is actually part of >>the implementation specification, and I can imagine other scenarios >>where case insensitive namespaces are "reasonable" even in the face >>of Python's excellent decision not to go this route. >> >>I wonder is this is actually the root of many requests to see this >>feature implemented? If not, it would be interesting to see what >>the competing desires really are. > > > Excellent observation. Perhaps the people who want this can get > together and write a PEP that explains the motivation and why the > alternatives don't work? (E.g. when you need case-insensitive > variable names, perhaps it's just as easy to preprocess the source > code?) I can think of two 'real world' use cases for this patch that I remember. 1) When I was consulting for a finite element modeling company that was using Python to manipulate _very_ large objects through a database. I don't recall the details, but basically they wanted to be able to control database connection lifetimes in a way that wasn't possible at the time (Python 1.4 or so, I think =). These days I think that proxies would probably be leveraged to provide the capability, but namespace lookup would still be useful -- one objection they had to using proxies is that they didn't want their users to think in terms of objects with attributes -- they wanted their users to say: a = b + c and have the system figure out at runtime whether to evaluate b, a DB connection needed to be established, or not. None of the alternatives I thought of at the time "worked" for various reasons. Sorry I'm so vague, it's been a long time. (I don't even remember how the problem was resolved). 2) The other case where I would have used this is in building a high-power interactive shell for scientific users doing modeling (think Matlab on steroids), where I wanted the interpreter to be able to resolve names at runtime from pickled states -- that way the memory load was dependent on actual usage, but the "impression" to the user would have been that the system "never forgot". In both of these cases, Guido would be right in saying that that's not Python, but that's somewhat irrelevant -- the end-users would have had a "better" user experience with the feature on, and other aspects of the system would have made it quite clear that this wasn't vanilla Python. I like the feature, not necessarily for inclusion in core Python but as a way of experimenting with alternate universes in a research mode: "What would a PHP-like 'isset()' feel like?" "Could we do something akin to Tcl's TclObj dual representations?" Name resolution has always felt like the least "hookable" of Python's mechanisms (as opposed to attribute lookup, exception handling, etc.), and I wonder what good things could come of making it more open to experimentation. --david PS: Thinking of PHP, the fact that I can look up variables defined in sessions or cookies or whatnot really really easily is one of the reasons I'm liking PHP more than Python for web programming these days. I don't know enough of the technical details of the web to know automatically where what data is stored and in what format -- and PHP doesn't make me. Maybe web folks would use the dict-subclass feature to emulate this aspect of PHP. From guido@python.org Tue Oct 29 18:29:16 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 29 Oct 2002 13:29:16 -0500 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: Your message of "Tue, 29 Oct 2002 10:20:28 PST." <3DBED16C.90801@ActiveState.com> References: <200210291243.g9TChoK24426@pcp02138704pcs.reston01.va.comcast.net> <3DBED16C.90801@ActiveState.com> Message-ID: <200210291829.g9TITGq32726@odiug.zope.com> > I can think of two 'real world' use cases for this patch that I > remember. > > 1) When I was consulting for a finite element modeling company that > was using Python to manipulate _very_ large objects through a > database. I don't recall the details, but basically they wanted to > be able to control database connection lifetimes in a way that > wasn't possible at the time (Python 1.4 or so, I think =). These > days I think that proxies would probably be leveraged to provide the > capability, but namespace lookup would still be useful -- one > objection they had to using proxies is that they didn't want their > users to think in terms of objects with attributes -- they wanted > their users to say: > > a = b + c > > and have the system figure out at runtime whether to evaluate b, a > DB connection needed to be established, or not. None of the > alternatives I thought of at the time "worked" for various reasons. > Sorry I'm so vague, it's been a long time. (I don't even remember > how the problem was resolved). > > 2) The other case where I would have used this is in building a > high-power interactive shell for scientific users doing modeling > (think Matlab on steroids), where I wanted the interpreter to be > able to resolve names at runtime from pickled states -- that way the > memory load was dependent on actual usage, but the "impression" to > the user would have been that the system "never forgot". > > In both of these cases, Guido would be right in saying that that's > not Python, but that's somewhat irrelevant -- the end-users would > have had a "better" user experience with the feature on, and other > aspects of the system would have made it quite clear that this > wasn't vanilla Python. I think (2) could be resolved using "ghosts" -- this is what Zope's persistency mechanism uses. The names must be present, but their content is loaded on demand. I expect that (1) probably could also be solved this way. > I like the feature, not necessarily for inclusion in core Python but > as a way of experimenting with alternate universes in a research > mode: "What would a PHP-like 'isset()' feel like?" "Could we do > something akin to Tcl's TclObj dual representations?" Name > resolution has always felt like the least "hookable" of Python's > mechanisms (as opposed to attribute lookup, exception handling, > etc.), and I wonder what good things could come of making it more > open to experimentation. That's a good question. > --david > > PS: Thinking of PHP, the fact that I can look up variables defined > in sessions or cookies or whatnot really really easily is one of the > reasons I'm liking PHP more than Python for web programming these > days. I don't know enough of the technical details of the web to > know automatically where what data is stored and in what format -- > and PHP doesn't make me. Maybe web folks would use the > dict-subclass feature to emulate this aspect of PHP. But why not make it an attribute lookup? That's how Zope (again :-) deals with this. Remember acquisition? :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From DavidA@ActiveState.com Tue Oct 29 19:05:06 2002 From: DavidA@ActiveState.com (David Ascher) Date: Tue, 29 Oct 2002 11:05:06 -0800 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec References: <200210291243.g9TChoK24426@pcp02138704pcs.reston01.va.comcast.net> <3DBED16C.90801@ActiveState.com> <200210291829.g9TITGq32726@odiug.zope.com> Message-ID: <3DBEDBE2.6060708@ActiveState.com> Guido van Rossum wrote: > I think (2) could be resolved using "ghosts" -- this is what Zope's > persistency mechanism uses. The names must be present, but their > content is loaded on demand. I expect that (1) probably could also be > solved this way. That assumes that you can "preload" all of the names at initialization, correct? IOW, that you know the contents of the namespace in advance. Not a huge limitation, but not the same as lazy ghost-creation on lookup. >>PS: Thinking of PHP, the fact that I can look up variables defined >>in sessions or cookies or whatnot really really easily is one of the >>reasons I'm liking PHP more than Python for web programming these >>days. I don't know enough of the technical details of the web to >>know automatically where what data is stored and in what format -- >>and PHP doesn't make me. Maybe web folks would use the >>dict-subclass feature to emulate this aspect of PHP. > > > But why not make it an attribute lookup? That's how Zope (again :-) > deals with this. Remember acquisition? :-) Yah, I remember. I'm not saying that one can't come up with a pythonic yet low-impact way of presenting web constructs to users. I guess I'm just expressing frustration with the fact that in PHP I don't really need to understand the web, whereas in Python I feel I have to. Seems wrong. =) --da From guido@python.org Tue Oct 29 19:11:13 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 29 Oct 2002 14:11:13 -0500 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: Your message of "Tue, 29 Oct 2002 11:05:06 PST." <3DBEDBE2.6060708@ActiveState.com> References: <200210291243.g9TChoK24426@pcp02138704pcs.reston01.va.comcast.net> <3DBED16C.90801@ActiveState.com> <200210291829.g9TITGq32726@odiug.zope.com> <3DBEDBE2.6060708@ActiveState.com> Message-ID: <200210291911.g9TJBDg13209@odiug.zope.com> > > I think (2) could be resolved using "ghosts" -- this is what Zope's > > persistency mechanism uses. The names must be present, but their > > content is loaded on demand. I expect that (1) probably could also be > > solved this way. > > That assumes that you can "preload" all of the names at > initialization, correct? IOW, that you know the contents of the > namespace in advance. Not a huge limitation, but not the same as > lazy ghost-creation on lookup. Correct. I don't see this as a huge limitation -- all "autoload" features that I'm familiar with (Emacs, Tcl; how about Perl?) require you to know at least the auto-loaded names in advance. (Also in my defence you only mentioned the cost of unpickling everything as an argument. :-) > >>PS: Thinking of PHP, the fact that I can look up variables defined > >>in sessions or cookies or whatnot really really easily is one of the > >>reasons I'm liking PHP more than Python for web programming these > >>days. I don't know enough of the technical details of the web to > >>know automatically where what data is stored and in what format -- > >>and PHP doesn't make me. Maybe web folks would use the > >>dict-subclass feature to emulate this aspect of PHP. > > > > > > But why not make it an attribute lookup? That's how Zope (again :-) > > deals with this. Remember acquisition? :-) > > Yah, I remember. I'm not saying that one can't come up with a > pythonic yet low-impact way of presenting web constructs to users. > I guess I'm just expressing frustration with the fact that in PHP I > don't really need to understand the web, whereas in Python I feel I > have to. Seems wrong. =) I guess this depends on which Python application you use. --Guido van Rossum (home page: http://www.python.org/~guido/) From dan@sidhe.org Tue Oct 29 19:19:29 2002 From: dan@sidhe.org (Dan Sugalski) Date: Tue, 29 Oct 2002 14:19:29 -0500 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: <200210291911.g9TJBDg13209@odiug.zope.com> References: <200210291243.g9TChoK24426@pcp02138704pcs.reston01.va.comcast.net> <3DBED16C.90801@ActiveState.com> <200210291829.g9TITGq32726@odiug.zope.com> <3DBEDBE2.6060708@ActiveState.com> <200210291911.g9TJBDg13209@odiug.zope.com> Message-ID: At 2:11 PM -0500 10/29/02, Guido van Rossum wrote: > > > I think (2) could be resolved using "ghosts" -- this is what Zope's >> > persistency mechanism uses. The names must be present, but their >> > content is loaded on demand. I expect that (1) probably could also be >> > solved this way. >> >> That assumes that you can "preload" all of the names at >> initialization, correct? IOW, that you know the contents of the >> namespace in advance. Not a huge limitation, but not the same as >> lazy ghost-creation on lookup. > >Correct. I don't see this as a huge limitation -- all "autoload" >features that I'm familiar with (Emacs, Tcl; how about Perl?) require >you to know at least the auto-loaded names in advance. (Also in my >defence you only mentioned the cost of unpickling everything as an >argument. :-) Perl doesn't require it. If a method call is made but that method name doesn't appear in the object's inheritance hierarchy, perl walks up the tree looking for a method named AUTOLOAD and if it finds one calls that. The AUTOLOAD method then has the opportunity to do something--satisfy the call, dispatch to a parent class, or pitch a fit. It's valid for an AUTOLOAD method to satisfy the request without instantiating a method for it. (In which case the next invocation of that method will end up back in AUTOLOAD) Works for subroutines too, though the hierarchy isn't supposed to be walked for those. (Older versions of perl would do it, but that's been deprecated) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido@python.org Tue Oct 29 19:26:26 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 29 Oct 2002 14:26:26 -0500 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: Your message of "Tue, 29 Oct 2002 14:19:29 EST." References: <200210291243.g9TChoK24426@pcp02138704pcs.reston01.va.comcast.net> <3DBED16C.90801@ActiveState.com> <200210291829.g9TITGq32726@odiug.zope.com> <3DBEDBE2.6060708@ActiveState.com> <200210291911.g9TJBDg13209@odiug.zope.com> Message-ID: <200210291926.g9TJQQs13694@odiug.zope.com> > >Correct. I don't see this as a huge limitation -- all "autoload" > >features that I'm familiar with (Emacs, Tcl; how about Perl?) require > >you to know at least the auto-loaded names in advance. (Also in my > >defence you only mentioned the cost of unpickling everything as an > >argument. :-) > > Perl doesn't require it. If a method call is made but that method > name doesn't appear in the object's inheritance hierarchy, perl walks > up the tree looking for a method named AUTOLOAD and if it finds one > calls that. The AUTOLOAD method then has the opportunity to do > something--satisfy the call, dispatch to a parent class, or pitch a > fit. It's valid for an AUTOLOAD method to satisfy the request without > instantiating a method for it. (In which case the next invocation of > that method will end up back in AUTOLOAD) Um, as long we're doing *method* lookup, Python's __getattr__ can do all that. > Works for subroutines too, though the hierarchy isn't supposed to be > walked for those. (Older versions of perl would do it, but that's > been deprecated) OK, so if there's a global AUTOLOAD and no global x, looking for x will call AUTOLOAD to satisfy x, right? Python doesn't do this, and this is what the "hookable namespace" feature is aiming at. --Guido van Rossum (home page: http://www.python.org/~guido/) From dan@sidhe.org Tue Oct 29 19:44:17 2002 From: dan@sidhe.org (Dan Sugalski) Date: Tue, 29 Oct 2002 14:44:17 -0500 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: <200210291926.g9TJQQs13694@odiug.zope.com> References: <200210291243.g9TChoK24426@pcp02138704pcs.reston01.va.comcast.net> <3DBED16C.90801@ActiveState.com> <200210291829.g9TITGq32726@odiug.zope.com> <3DBEDBE2.6060708@ActiveState.com> <200210291911.g9TJBDg13209@odiug.zope.com> <200210291926.g9TJQQs13694@odiug.zope.com> Message-ID: At 2:26 PM -0500 10/29/02, Guido van Rossum wrote: > > >Correct. I don't see this as a huge limitation -- all "autoload" >> >features that I'm familiar with (Emacs, Tcl; how about Perl?) require >> >you to know at least the auto-loaded names in advance. (Also in my >> >defence you only mentioned the cost of unpickling everything as an >> >argument. :-) >> >> Perl doesn't require it. If a method call is made but that method >> name doesn't appear in the object's inheritance hierarchy, perl walks >> up the tree looking for a method named AUTOLOAD and if it finds one >> calls that. The AUTOLOAD method then has the opportunity to do >> something--satisfy the call, dispatch to a parent class, or pitch a >> fit. It's valid for an AUTOLOAD method to satisfy the request without >> instantiating a method for it. (In which case the next invocation of >> that method will end up back in AUTOLOAD) > >Um, as long we're doing *method* lookup, Python's __getattr__ can do >all that. Sorry--it's a twisty conversation of little passages. I got somewhat turned around. > > Works for subroutines too, though the hierarchy isn't supposed to be >> walked for those. (Older versions of perl would do it, but that's >> been deprecated) > >OK, so if there's a global AUTOLOAD and no global x, looking for x >will call AUTOLOAD to satisfy x, right? Python doesn't do this, and >this is what the "hookable namespace" feature is aiming at. If there's an AUTOLOAD in a package, no X in that package, and you call X, then that package's AUTOLOAD will get called instead, yes. So if you had: print(Foo::Bar()); to print the output of the subroutine Bar in the package Foo, and there was no subroutine Bar in the package Foo but there *is* a subroutine AUTOLOAD in the package Foo, then Foo's AUTOLOAD will get called instead. Appropriate variables are filled in so that AUTOLOAD method can tell what sub it was called for. Only works for subs, not for global variables. (That's for perl 6, I expect) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From pobrien@orbtech.com Tue Oct 29 22:21:58 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Tue, 29 Oct 2002 16:21:58 -0600 Subject: [Python-Dev] Keyword module cruft question Message-ID: <200210291621.58649.pobrien@orbtech.com> I'm working on an article about Python introspection and came across some= thing=20 I consider a blemish. I thought I'd ask about it here and see how you fol= ks=20 feel about it. BTW, I'm looking at Python 2.2.2 on Linux. If you look at the keyword module, you'll see that it has a keyword attri= bute: >>> import keyword >>> dir(keyword) ['__all__', '__builtins__', '__doc__', '__file__', '__name__', 'iskeyword= ',=20 'keyword', 'kwdict', 'kwlist', 'main'] >>> keyword.keyword 'yield' >>>=20 Basically, the keyword attribute is just the last value from this process= : kwdict =3D {} for keyword in kwlist: kwdict[keyword] =3D 1 So it is a bit of extraneous cruft, I believe. No big deal, and nothing t= hat a del keyword couldn't fix. But it does look sloppy when you are writing about introspe= ction=20 and you see an attribute that has no real value. I wish it weren't there,= as=20 I'd rather not have to explain it. But it got me thinking whether other=20 modules in the standard library have this issue, and whether this is=20 something that should be cleaned up. Thoughts? --=20 Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From guido@python.org Tue Oct 29 22:41:57 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 29 Oct 2002 17:41:57 -0500 Subject: [Python-Dev] Keyword module cruft question In-Reply-To: Your message of "Tue, 29 Oct 2002 16:21:58 CST." <200210291621.58649.pobrien@orbtech.com> References: <200210291621.58649.pobrien@orbtech.com> Message-ID: <200210292241.g9TMfvE16004@odiug.zope.com> > I'm working on an article about Python introspection and came across > something I consider a blemish. I thought I'd ask about it here and > see how you folks feel about it. BTW, I'm looking at Python 2.2.2 on > Linux. > > If you look at the keyword module, you'll see that it has a keyword > attribute: > > >>> import keyword > >>> dir(keyword) > ['__all__', '__builtins__', '__doc__', '__file__', '__name__', 'iskeyword', > 'keyword', 'kwdict', 'kwlist', 'main'] > >>> keyword.keyword > 'yield' > >>> > > Basically, the keyword attribute is just the last value from this > process: > > kwdict = {} > for keyword in kwlist: > kwdict[keyword] = 1 > > So it is a bit of extraneous cruft, I believe. No big deal, and > nothing that a > > del keyword > > couldn't fix. But it does look sloppy when you are writing about > introspection and you see an attribute that has no real value. I > wish it weren't there, as I'd rather not have to explain it. But it > got me thinking whether other modules in the standard library have > this issue, and whether this is something that should be cleaned > up. Thoughts? Lots of modules have attributes that are cruft according to this definition. I personally see no reason to get rid of all these -- they're harmless and "from M import *" doesn't import them. Feel free to hold your own code to a higher standard though. ;-) --Guido van Rossum (home page: http://www.python.org/~guido/) From Jack.Jansen@oratrix.com Tue Oct 29 22:43:52 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Tue, 29 Oct 2002 23:43:52 +0100 Subject: [Python-Dev] Patch to use dict subclasses in eval(), exec In-Reply-To: <200210291829.g9TITGq32726@odiug.zope.com> Message-ID: If at some point it's decided that this patch is a good idea I think we can get rid of the performance hit in the normal case by duplicating eval_code2() with some preprocessor magic. eval_code2() would then call out to eval_code2_non_dict() if the globals weren't in a dict. Hmm, I wonder whether there are other situations where this idea would work too, i.e. situations where eval_code2() could decide at startup that it could take a fast path... -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From martin@v.loewis.de Tue Oct 29 23:02:16 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 30 Oct 2002 00:02:16 +0100 Subject: [Python-Dev] Keyword module cruft question In-Reply-To: <200210291621.58649.pobrien@orbtech.com> References: <200210291621.58649.pobrien@orbtech.com> Message-ID: "Patrick K. O'Brien" writes: > But it does look sloppy when you are writing about introspection and > you see an attribute that has no real value. I wish it weren't > there, as I'd rather not have to explain it. But it got me thinking > whether other modules in the standard library have this issue, and > whether this is something that should be cleaned up. Thoughts? Normally, this is fixed with __all__: The module may have arbitrary variables; the interface is only those identifiers mentioned in __all__ (in absence of __all__, its only those identifiers which are documented). So I don't think this is an issue, in general. In a specific case, there is nothing wrong with fixing it. Regards, Martin From pobrien@orbtech.com Tue Oct 29 23:27:19 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Tue, 29 Oct 2002 17:27:19 -0600 Subject: [Python-Dev] Keyword module cruft question In-Reply-To: References: <200210291621.58649.pobrien@orbtech.com> Message-ID: <200210291727.19495.pobrien@orbtech.com> On Tuesday 29 October 2002 05:02 pm, Martin v. Loewis wrote: > "Patrick K. O'Brien" writes: > > But it does look sloppy when you are writing about introspection and > > you see an attribute that has no real value. I wish it weren't > > there, as I'd rather not have to explain it. But it got me thinking > > whether other modules in the standard library have this issue, and > > whether this is something that should be cleaned up. Thoughts? > > Normally, this is fixed with __all__: The module may have arbitrary > variables; the interface is only those identifiers mentioned in > __all__ (in absence of __all__, its only those identifiers which are > documented). Well, in this case, keyword's __all__ looks like: __all__ =3D ["iskeyword"] Which leaves out keyword.kwlist, a pretty useful attribute. > So I don't think this is an issue, in general. In a specific case, > there is nothing wrong with fixing it. I'd also suggest that __all__ as a determiner of a module's api is severe= ly=20 underdocumented. Maybe I just picked the wrong module to start off my article. ;-) --=20 Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From pobrien@orbtech.com Tue Oct 29 23:52:20 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Tue, 29 Oct 2002 17:52:20 -0600 Subject: [Python-Dev] Keyword module cruft question In-Reply-To: <200210292241.g9TMfvE16004@odiug.zope.com> References: <200210291621.58649.pobrien@orbtech.com> <200210292241.g9TMfvE16004@odiug.zope.com> Message-ID: <200210291752.20822.pobrien@orbtech.com> On Tuesday 29 October 2002 04:41 pm, Guido van Rossum wrote: > Lots of modules have attributes that are cruft according to this > definition. I personally see no reason to get rid of all these -- > they're harmless and "from M import *" doesn't import them. Feel free > to hold your own code to a higher standard though. ;-) I understand what you are saying, and respectfully disagree. Extraneous=20 attributes are harmful to the extent that one has to know to ignore them.= =20 Knowing what to ignore is difficult when learning Python or working with = an=20 unfamiliar module. And extraneous attributes are cumbersome when making u= se=20 of Python's introspection features, such as dir(object), or when using a=20 Python shell or editor that offers autocompletion. And, finally, they get= in=20 the way while writing an article about Python introspection. ;-) Also, little things like this make it more challenging to present Python = in=20 its most positive light for a general audience, featuring characteristics= of=20 the language that are supposed to show how it is a better choice than oth= er=20 languages. I'm not saying that having a few extraneous attributes is a hu= ge=20 flaw. And you know how much of a fan of Python I am. But anything that ha= s to=20 be "brushed under the rug" can create doubt among those who aren't alread= y=20 part of the Python community. And I don't think a higher standard would h= urt=20 the standard library. Just something to think about. --=20 Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From martin@v.loewis.de Wed Oct 30 00:14:41 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 30 Oct 2002 01:14:41 +0100 Subject: [Python-Dev] Incorporating bsddb3 Message-ID: [Since the last message triggered no response, I'm retrying.] I have changes in my sandbox to incorporate bsddb3 into Python proper. This essentially is a copy of bsddb3 3.4.0, with the following modifications: - the extension module is called _bsddb (not bsddb3._db) - the package is called bsddb (not bsddb3) Both test_bsddb and test_anydbm continue to work unmodified. Is it ok to commit these changes? Regards, Martin From guido@python.org Wed Oct 30 01:38:54 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 29 Oct 2002 20:38:54 -0500 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: Your message of "Wed, 30 Oct 2002 01:14:41 +0100." References: Message-ID: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> > I have changes in my sandbox to incorporate bsddb3 into Python > proper. This essentially is a copy of bsddb3 3.4.0, with the following > modifications: > - the extension module is called _bsddb (not bsddb3._db) > - the package is called bsddb (not bsddb3) > > Both test_bsddb and test_anydbm continue to work unmodified. What happens if you don't have the software installed to make this work, but you do have what it takes to make the *old* bsddb module work? I presume that one bites the dust now? --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Wed Oct 30 05:02:11 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 30 Oct 2002 06:02:11 +0100 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > What happens if you don't have the software installed to make this > work, but you do have what it takes to make the *old* bsddb module > work? I presume that one bites the dust now? That would be bsddb2, which is still supported in the old bsddbmodule.c, but not bsddb3. I would propose to continue shipping bsddbmodule.c. Building and using that would then require manual intervention. Regards, Martin From guido@python.org Wed Oct 30 05:12:35 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 30 Oct 2002 00:12:35 -0500 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: Your message of "Wed, 30 Oct 2002 06:02:11 +0100." References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> > > What happens if you don't have the software installed to make this > > work, but you do have what it takes to make the *old* bsddb module > > work? I presume that one bites the dust now? > > That would be bsddb2, which is still supported in the old > bsddbmodule.c, but not bsddb3. > > I would propose to continue shipping bsddbmodule.c. Building and using > that would then require manual intervention. Perhaps it could be made into an alternative for _bsddb to make that simpler (someone could even contribute code to let setup.py decide). What's the license on the bsddb3 Python extension? What's the license on the BerkeleyDB code from Sleepycat? Can we legally distribute RPMs or other binaries containing it? (I thought there were some restrictions that make it not open source.) --Guido van Rossum (home page: http://www.python.org/~guido/) From python@rcn.com Wed Oct 30 05:19:44 2002 From: python@rcn.com (Raymond Hettinger) Date: Wed, 30 Oct 2002 00:19:44 -0500 Subject: [Python-Dev] Keyword module cruft question References: <200210291621.58649.pobrien@orbtech.com> <200210291727.19495.pobrien@orbtech.com> Message-ID: <006501c27fd3$f605d6c0$13d8accf@oemcomputer> [Patrick K. O'Brien] > Well, in this case, keyword's __all__ looks like: > > __all__ = ["iskeyword"] > > Which leaves out keyword.kwlist, a pretty useful attribute. Fixed. > I'd also suggest that __all__ as a determiner of a module's api is severely > underdocumented. Agreed. Please file a documentation bug report and assign to me. Raymond Hettinger From martin@v.loewis.de Wed Oct 30 05:51:41 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 30 Oct 2002 06:51:41 +0100 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > Perhaps it could be made into an alternative for _bsddb to make that > simpler (someone could even contribute code to let setup.py decide). Sure. > What's the license on the bsddb3 Python extension? See below. > What's the license on the BerkeleyDB code from Sleepycat? There are two licenses: One that they call the "open source license", see http://www.sleepycat.com/license.net There is also a commercial license. > Can we legally distribute RPMs or other binaries containing it? (I > thought there were some restrictions that make it not open source.) It depends. This is the condition: # Redistributions in any form must be accompanied by information on # how to obtain complete source code for the DB software and any # accompanying software that uses the DB software. The source code # must either be included in the distribution or be available for no # more than the cost of distribution plus a nominal fee, and must be # freely redistributable under reasonable conditions. So distributing Python itself should be no problem. It appears that the Pythonlabs distribution still uses DB 1.85. Don't tell that Skip :-) He believes that this version has serious flaws that can lead to data corruption... Regards, Martin From martin@v.loewis.de Wed Oct 30 06:00:54 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 30 Oct 2002 07:00:54 +0100 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> Message-ID: martin@v.loewis.de (Martin v. Loewis) writes: > See below. Actually, look here. /*---------------------------------------------------------------------- Copyright (c) 1999-2001, Digital Creations, Fredericksburg, VA, USA and Andrew Kuchling. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: o Redistributions of source code must retain the above copyright notice, this list of conditions, and the disclaimer that follows. o Redistributions in binary form must reproduce the above copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution. o Neither the name of Digital Creations nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS AND CONTRIBUTORS *AS IS* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ------------------------------------------------------------------------*/ It's all yours :-) Regards, Martin From mwh@python.net Wed Oct 30 13:00:28 2002 From: mwh@python.net (Michael Hudson) Date: 30 Oct 2002 13:00:28 +0000 Subject: [Python-Dev] metaclass insanity Message-ID: <2mfzuoytcj.fsf@starship.python.net> For moderately nefarious reasons[1] I've being trying to write a metaclass whose instances have writable __bases__. This in itself isn't so hard, but having assigments to __bases__ "do the right thing" has eluded me, basically because I can't seem to affect the mro. Here's what I have so far: def base_getter(klass): return klass.__priv_bases__ def base_setter(klass, v): klass.__priv_bases__ = v class MetaMutableBases(type): def __new__(self, name, bases, ns): t = type.__new__(self, name, bases, ns) t.__priv_bases__ = type.__dict__["__bases__"].__get__(self) return t __bases__ = property(base_getter, base_setter) def mro(self): if hasattr(self, "__priv_bases__"): return type("noname", self.__priv_bases__, {}).__mro__ else: return type.mro(self) I've tried variations on the same, but haven't got anything working yet. Anyone have any ideas? I think my brain's exploded. And I realise this is in some sense more appropriate for c.l.py, but I think my odds of a good answer are vastly higher here. Cheers, M. [1] in http://groups.google.com/groups?selm=m2r8e99l0b.fsf%40python.net I posted a metaclass that attempts to automatically upgrade instances of its instances on redefinition. Ideally, this would also frob its subclasses so they now inherited from redefined class. I'm aware this can go wrong in various ways, but I think it would be pretty handy in a large range of situations. -- This is not to say C++ = bad, Lisp = good. It's to say C++ = bad irrespective of everything else. -- Alain Picard, comp.lang.lisp From guido@python.org Wed Oct 30 13:45:59 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 30 Oct 2002 08:45:59 -0500 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: Your message of "Wed, 30 Oct 2002 07:00:54 +0100." References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210301345.g9UDjxb21402@pcp02138704pcs.reston01.va.comcast.net> > Copyright (c) 1999-2001, Digital Creations, Fredericksburg, VA, USA > and Andrew Kuchling. All rights reserved. Now I'm confused. I thought this was Robun Dunn's bsddb3 module? Is there another? --Guido van Rossum (home page: http://www.python.org/~guido/) From akuchlin@mems-exchange.org Wed Oct 30 14:06:22 2002 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Wed, 30 Oct 2002 09:06:22 -0500 Subject: [Python-Dev] Re: Incorporating bsddb3 Message-ID: GvR wrote: >Now I'm confused. I thought this was Robun Dunn's bsddb3 module? Is >there another? It's the same one. Robin Dunn's original module used SWIG to wrap the BerkeleyDB API, and I offered to rewrite the interface as a hand-written extension module, which is why my name turns up. (Adding my name to the copyright seemed unjustified to me at the time, though; writing the extension was mostly a trivial process of rearranging Robin's SWIG definitions with some Emacs/elisp work.) --amk (www.amk.ca) JULIET: O Romeo, Romeo, wherefore art thou Romeo? -- _Romeo and Juliet_, II, ii From jacobs@penguin.theopalgroup.com Wed Oct 30 14:15:52 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Wed, 30 Oct 2002 09:15:52 -0500 (EST) Subject: [Python-Dev] metaclass insanity In-Reply-To: <2mfzuoytcj.fsf@starship.python.net> Message-ID: On 30 Oct 2002, Michael Hudson wrote: > For moderately nefarious reasons[1] I've being trying to write a > metaclass whose instances have writable __bases__. This in itself > isn't so hard, but having assigments to __bases__ "do the right thing" > has eluded me, basically because I can't seem to affect the mro. The mro is an internal data structure of new-style classes, so redefining mro() doesn't change the values used. Here is my (non-working version) that attempts to re-assign the class of an object, although it fails on a layout violation with Python 2.2.2. def base_getter(cls): return cls.__my_bases__ def base_setter(cls,bases): if not bases: bases = (object,) metaclass = getattr(cls, '__metaclass__', type) new_cls = metaclass(cls.__name__, bases, dict(cls.__dict__)) cls.__class__ = new_cls class MetaBase(type): __bases__ = property(base_getter,base_setter) def __new__(cls, name, bases, ns): ns['__my_bases__'] = tuple(bases) return super(MetaBase, cls).__new__(cls, name, bases, ns) class Foo(object): __metaclass__ = MetaBase class Baz(object): pass Foo.__bases__ = Foo.__bases__ + (Baz,) Which results in: TypeError: __class__ assignment: 'Foo' object layout differs from 'MetaBase' I haven't looked into why this is being flagged as a layout error, though my first instinct is to say that the check is too conservative in this case. I'll think about it more and dig into the code. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From barry@python.org Wed Oct 30 14:17:19 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 30 Oct 2002 09:17:19 -0500 Subject: [Python-Dev] Incorporating bsddb3 References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> <200210301345.g9UDjxb21402@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15807.59887.992769.142566@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: >> Copyright (c) 1999-2001, Digital Creations, Fredericksburg, VA, >> USA and Andrew Kuchling. All rights reserved. GvR> Now I'm confused. I thought this was Robun Dunn's bsddb3 GvR> module? Is there another? ZC paid Robin to develop bsddb3. Early on when we were developing the Berkeley storages for ZODB, we specifically asked Sleepycat about redistribution and they said that there would be no legal problem to us redistributing source or binaries with Python, because Python is also open source. I wouldn't take my word for that though -- I'd want to get that in writing -- if we decided to distribute the library. I'm not sure we'd want to distribute either the source or binary though. They're fairly large. If we did, I'd recommend BerkeleyDB 4.0.14, not the latest version, because bsddb3 doesn't yet build on 4.1.24 because of some incompatible API changes, IIUC. Note that bsddb3's Windows distro comes with the BerkeleyDB dll's, I beleive. Martin, what do you plan to do about the pybsddb test suite? It's also fairly large. If you do add that, I'd suggest not running it by default, but only with a -resource flag. Then there's the documentation and project management issues to talk about. Maybe we include none of that, but simply "import" the code into Python and re-sync whenever it's appropriate. -Barry From pobrien@orbtech.com Wed Oct 30 14:23:58 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Wed, 30 Oct 2002 08:23:58 -0600 Subject: [Python-Dev] Keyword module cruft question In-Reply-To: <006501c27fd3$f605d6c0$13d8accf@oemcomputer> References: <200210291621.58649.pobrien@orbtech.com> <200210291727.19495.pobrien@orbtech.com> <006501c27fd3$f605d6c0$13d8accf@oemcomputer> Message-ID: <200210300823.58866.pobrien@orbtech.com> On Tuesday 29 October 2002 11:19 pm, Raymond Hettinger wrote: > > I'd also suggest that __all__ as a determiner of a module's api is > > severely underdocumented. > > Agreed. Please file a documentation bug report and assign to me. Done. SF# 631055. Thanks. --=20 Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From mwh@python.net Wed Oct 30 14:36:20 2002 From: mwh@python.net (Michael Hudson) Date: 30 Oct 2002 14:36:20 +0000 Subject: [Python-Dev] metaclass insanity In-Reply-To: Kevin Jacobs's message of "Wed, 30 Oct 2002 09:15:52 -0500 (EST)" References: Message-ID: <2md6psyowr.fsf@starship.python.net> Kevin Jacobs writes: > On 30 Oct 2002, Michael Hudson wrote: > > For moderately nefarious reasons[1] I've being trying to write a > > metaclass whose instances have writable __bases__. This in itself > > isn't so hard, but having assigments to __bases__ "do the right thing" > > has eluded me, basically because I can't seem to affect the mro. > > The mro is an internal data structure of new-style classes, so redefining > mro() doesn't change the values used. Yeah, I noticed that eventually. > Here is my (non-working version) that > attempts to re-assign the class of an object, although it fails on a layout > violation with Python 2.2.2. > > def base_getter(cls): > return cls.__my_bases__ > > def base_setter(cls,bases): > if not bases: > bases = (object,) > metaclass = getattr(cls, '__metaclass__', type) > new_cls = metaclass(cls.__name__, bases, dict(cls.__dict__)) > cls.__class__ = new_cls > > class MetaBase(type): > __bases__ = property(base_getter,base_setter) > > def __new__(cls, name, bases, ns): > ns['__my_bases__'] = tuple(bases) > return super(MetaBase, cls).__new__(cls, name, bases, ns) > > class Foo(object): > __metaclass__ = MetaBase > class Baz(object): pass > > Foo.__bases__ = Foo.__bases__ + (Baz,) I don't think this has a hope of working does it? This approach would need to rebind "Foo" in the last line, no? > Which results in: > TypeError: __class__ assignment: 'Foo' object layout differs from 'MetaBase' > > I haven't looked into why this is being flagged as a layout error, though > my first instinct is to say that the check is too conservative in this case. > I'll think about it more and dig into the code. I think the error message gives it away: in "cls.__class__ = new_cls", cls.__class__ is MetaBase, new_cls is the new Foo. I guess one way of doing this would be to reimplement mro resolution and so on in Python, which would be annoying and inefficient. Hmm. Cheers, M. -- [3] Modem speeds being what they are, large .avi files were generally downloaded to the shell server instead[4]. [4] Where they were usually found by the technical staff, and burned to CD. -- Carlfish, asr From guido@python.org Wed Oct 30 15:31:32 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 30 Oct 2002 10:31:32 -0500 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: Your message of "Wed, 30 Oct 2002 06:51:41 +0100." References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200210301531.g9UFVW918144@odiug.zope.com> > > What's the license on the BerkeleyDB code from Sleepycat? > > There are two licenses: One that they call the "open source license", > see > > http://www.sleepycat.com/license.net > > There is also a commercial license. > > > Can we legally distribute RPMs or other binaries containing it? (I > > thought there were some restrictions that make it not open source.) > > It depends. This is the condition: > > # Redistributions in any form must be accompanied by information on > # how to obtain complete source code for the DB software and any > # accompanying software that uses the DB software. The source code > # must either be included in the distribution or be available for no > # more than the cost of distribution plus a nominal fee, and must be > # freely redistributable under reasonable conditions. > > So distributing Python itself should be no problem. It's similar to GPL's "copyleft". I think it's no different from what we do with e.g. GNU readline, so I think it's okay. Redistributors of Python in binary form will have to beware though. I wonder if we're on thin ice with the RPMs (since we don't clarify any of this)? --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Wed Oct 30 15:46:47 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 30 Oct 2002 16:46:47 +0100 Subject: [Python-Dev] Incorporating bsddb3 References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> <200210301531.g9UFVW918144@odiug.zope.com> Message-ID: <3DBFFEE7.6080304@lemburg.com> Guido van Rossum wrote: >>>What's the license on the BerkeleyDB code from Sleepycat? >> >>There are two licenses: One that they call the "open source license", >>see >> >>http://www.sleepycat.com/license.net >> >>There is also a commercial license. >> >> >>>Can we legally distribute RPMs or other binaries containing it? (I >>>thought there were some restrictions that make it not open source.) >> >>It depends. This is the condition: >> >># Redistributions in any form must be accompanied by information on >># how to obtain complete source code for the DB software and any >># accompanying software that uses the DB software. The source code >># must either be included in the distribution or be available for no >># more than the cost of distribution plus a nominal fee, and must be >># freely redistributable under reasonable conditions. >> >>So distributing Python itself should be no problem. > > > It's similar to GPL's "copyleft". I think it's no different from what > we do with e.g. GNU readline, so I think it's okay. Redistributors of > Python in binary form will have to beware though. I wonder if we're > on thin ice with the RPMs (since we don't clarify any of this)? ... but we're not including readline in the standard Python distribution. Why would we want to include the database code itself in Python ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Wed Oct 30 16:04:12 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 30 Oct 2002 11:04:12 -0500 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: Your message of "Wed, 30 Oct 2002 16:46:47 +0100." <3DBFFEE7.6080304@lemburg.com> References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> <200210301531.g9UFVW918144@odiug.zope.com> <3DBFFEE7.6080304@lemburg.com> Message-ID: <200210301604.g9UG4CF18245@odiug.zope.com> > > It's similar to GPL's "copyleft". I think it's no different from what > > we do with e.g. GNU readline, so I think it's okay. Redistributors of > > Python in binary form will have to beware though. I wonder if we're > > on thin ice with the RPMs (since we don't clarify any of this)? > > ... but we're not including readline in the standard Python > distribution. Why would we want to include the database code > itself in Python ? Not in the source distro, but it will end up in the RPMs and other binary distros. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 30 16:44:36 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 30 Oct 2002 11:44:36 -0500 Subject: [Python-Dev] metaclass insanity In-Reply-To: Your message of "Wed, 30 Oct 2002 14:36:20 GMT." <2md6psyowr.fsf@starship.python.net> References: <2md6psyowr.fsf@starship.python.net> Message-ID: <200210301644.g9UGiaZ18410@odiug.zope.com> > I guess one way of doing this would be to reimplement mro resolution > and so on in Python, which would be annoying and inefficient. Hmm. Alas, yes. You'd have to be able to write a class's tp_mro slot (corresponding to the __mro__ attribute), but there's no way to do that in Python, because it is too easy to cause mistakes. E.g. all C code that currently uses tp_mro assumes that it is a tuple whose items are either types or classic classes. The best thing to do would perhaps to make __mro__ assignable, but with a check that ensures the above constraint. I think I'd take a patch for that. I'd also take a patch for assignable __bases__. Note that there are constraints between __bases__ and __base__. I'd also take a patch for assignable __name__. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Wed Oct 30 16:46:19 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 30 Oct 2002 17:46:19 +0100 Subject: [Python-Dev] Incorporating bsddb3 References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> <200210301531.g9UFVW918144@odiug.zope.com> <3DBFFEE7.6080304@lemburg.com> <200210301604.g9UG4CF18245@odiug.zope.com> Message-ID: <3DC00CDB.9060109@lemburg.com> Guido van Rossum wrote: >>>It's similar to GPL's "copyleft". I think it's no different from what >>>we do with e.g. GNU readline, so I think it's okay. Redistributors of >>>Python in binary form will have to beware though. I wonder if we're >>>on thin ice with the RPMs (since we don't clarify any of this)? >> >>... but we're not including readline in the standard Python >>distribution. Why would we want to include the database code >>itself in Python ? > > Not in the source distro, but it will end up in the RPMs and other > binary distros. But that would violate the Sleepycat license; we are not shipping readline or a module which statically links against readline in the binary distros for the same reason. As I understand, the BSD version shipped with Python so far is using a standard BSD style license, so there's no problem with it. However, Sleepycat added a copyleft kind of restriction to the BSD license which makes this troublesome, esp. since the restriction is not at all clear about what "use of the DB software" really means. Their web-page on this is: http://www.sleepycat.com/licensing.html Also, the Sleepycat license is not compatible with the Python license in the sense that the Python license is far less restrictive than the Sleepycat one. We'd have to make it very clear that we have integrated third-party code into the archive which falls under a much more restrictive license. (FWIW, It would actually be wise to summarize all the different licenses in the Python distro somewhere...) So back to my original question: why go through all these troubles ? BTW, Sleepycat ships Berkley DB 4.1, wouldn't it be wise to update the wrapper module to that version ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From barry@python.org Wed Oct 30 17:33:28 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 30 Oct 2002 12:33:28 -0500 Subject: [Python-Dev] Incorporating bsddb3 References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> <200210301531.g9UFVW918144@odiug.zope.com> <3DBFFEE7.6080304@lemburg.com> <200210301604.g9UG4CF18245@odiug.zope.com> <3DC00CDB.9060109@lemburg.com> Message-ID: <15808.6120.271065.946000@gargle.gargle.HOWL> >>>>> "MAL" == M writes: >> Not in the source distro, but it will end up in the RPMs and >> other binary distros. MAL> But that would violate the Sleepycat license; we are not MAL> shipping readline or a module which statically links against MAL> readline in the binary distros for the same reason. Again, when we talked to Sleepycat a year and a half ago, they said it would be ok to distribute the library with Python. IIRC, even a binary distro would be fine. But I think a simple email to them would clear the issue up. MAL> BTW, Sleepycat ships Berkley DB 4.1, wouldn't it be wise to MAL> update the wrapper module to that version ? Because there are problems: http://sourceforge.net/mailarchive/forum.php?thread_id=1206938&forum_id=4362 -Barry From guido@python.org Wed Oct 30 17:45:08 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 30 Oct 2002 12:45:08 -0500 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: Your message of "Wed, 30 Oct 2002 17:46:19 +0100." <3DC00CDB.9060109@lemburg.com> References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> <200210301531.g9UFVW918144@odiug.zope.com> <3DBFFEE7.6080304@lemburg.com> <200210301604.g9UG4CF18245@odiug.zope.com> <3DC00CDB.9060109@lemburg.com> Message-ID: <200210301745.g9UHj8t18711@odiug.zope.com> > >>>It's similar to GPL's "copyleft". I think it's no different from what > >>>we do with e.g. GNU readline, so I think it's okay. Redistributors of > >>>Python in binary form will have to beware though. I wonder if we're > >>>on thin ice with the RPMs (since we don't clarify any of this)? > >> > >>... but we're not including readline in the standard Python > >>distribution. Why would we want to include the database code > >>itself in Python ? > > > > Not in the source distro, but it will end up in the RPMs and other > > binary distros. > > But that would violate the Sleepycat license; we are not shipping > readline or a module which statically links against readline > in the binary distros for the same reason. It only violates the Sleepycat license if (a) we don't include that license with the RPM or (b) we include closed-source code in that RPM. We're violating only (a) AFAIK and that's easily fixed. > As I understand, the BSD version shipped with Python so far > is using a standard BSD style license, so there's no problem > with it. However, Sleepycat added a copyleft kind of restriction > to the BSD license which makes this troublesome, esp. since the > restriction is not at all clear about what "use of the DB software" > really means. Their web-page on this is: > > http://www.sleepycat.com/licensing.html > > Also, the Sleepycat license is not compatible with the Python > license in the sense that the Python license is far less > restrictive than the Sleepycat one. We'd have to make it very > clear that we have integrated third-party code into the > archive which falls under a much more restrictive license. > (FWIW, It would actually be wise to summarize all the different > licenses in the Python distro somewhere...) > > So back to my original question: why go through all these > troubles ? We're not planning to incorporate Sleepycat code in the python source distribution. But if setup.py builds bsddb if it finds the parts, it's hard to keep the Sleepycat binaries in the RPMs. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh@python.net Wed Oct 30 18:17:26 2002 From: mwh@python.net (Michael Hudson) Date: 30 Oct 2002 18:17:26 +0000 Subject: [Python-Dev] metaclass insanity In-Reply-To: Guido van Rossum's message of "Wed, 30 Oct 2002 11:44:36 -0500" References: <2md6psyowr.fsf@starship.python.net> <200210301644.g9UGiaZ18410@odiug.zope.com> Message-ID: <2my98faj0p.fsf@starship.python.net> Guido van Rossum writes: > > I guess one way of doing this would be to reimplement mro resolution > > and so on in Python, which would be annoying and inefficient. Hmm. > > Alas, yes. You'd have to be able to write a class's tp_mro slot > (corresponding to the __mro__ attribute), but there's no way to do > that in Python, because it is too easy to cause mistakes. E.g. all > C code that currently uses tp_mro assumes that it is a tuple whose > items are either types or classic classes. > > The best thing to do would perhaps to make __mro__ assignable, but > with a check that ensures the above constraint. I think I'd take a > patch for that. Shouldn't be too hard. > I'd also take a patch for assignable __bases__. Note that there are > constraints between __bases__ and __base__. Should assigning to __bases__ automatically tweak __mro__ and __base__? Guess so. What would assigning to __base__ do in isolation? Perhaps that shouldn't be writeable. > I'd also take a patch for assignable __name__. This is practically a one-liner, isn't it? Not hard, anyway. And there was me wondering what I was going to do this evening. Cheers, M. -- Now this is what I don't get. Nobody said absolutely anything bad about anything. Yet it is always possible to just pull random flames out of ones ass. -- http://www.advogato.org/person/vicious/diary.html?start=60 From mal@lemburg.com Wed Oct 30 18:19:57 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 30 Oct 2002 19:19:57 +0100 Subject: [Python-Dev] Incorporating bsddb3 References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> <200210301531.g9UFVW918144@odiug.zope.com> <3DBFFEE7.6080304@lemburg.com> <200210301604.g9UG4CF18245@odiug.zope.com> <3DC00CDB.9060109@lemburg.com> <200210301745.g9UHj8t18711@odiug.zope.com> Message-ID: <3DC022CD.5010607@lemburg.com> Guido van Rossum wrote: >>>Not in the source distro, but it will end up in the RPMs and other >>>binary distros. >> >>But that would violate the Sleepycat license; we are not shipping >>readline or a module which statically links against readline >>in the binary distros for the same reason. > > It only violates the Sleepycat license if (a) we don't include > that license with the RPM or (b) we include closed-source code in that > RPM. We're violating only (a) AFAIK and that's easily fixed. Ok. >>As I understand, the BSD version shipped with Python so far >>is using a standard BSD style license, so there's no problem >>with it. However, Sleepycat added a copyleft kind of restriction >>to the BSD license which makes this troublesome, esp. since the >>restriction is not at all clear about what "use of the DB software" >>really means. Their web-page on this is: >> >> http://www.sleepycat.com/licensing.html >> >>Also, the Sleepycat license is not compatible with the Python >>license in the sense that the Python license is far less >>restrictive than the Sleepycat one. We'd have to make it very >>clear that we have integrated third-party code into the >>archive which falls under a much more restrictive license. >>(FWIW, It would actually be wise to summarize all the different >>licenses in the Python distro somewhere...) >> >>So back to my original question: why go through all these >>troubles ? > > We're not planning to incorporate Sleepycat code in the python > source distribution. But if setup.py builds bsddb if it finds the > parts, it's hard to keep the Sleepycat binaries in the RPMs. It's not like it's impossible, though :-). Ideal would be to have the Python module link against the shared lib of the Sleepycat DB (if that's possible). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From barry@python.org Wed Oct 30 18:22:24 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 30 Oct 2002 13:22:24 -0500 Subject: [Python-Dev] Incorporating bsddb3 References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> <200210301531.g9UFVW918144@odiug.zope.com> <3DBFFEE7.6080304@lemburg.com> <200210301604.g9UG4CF18245@odiug.zope.com> <3DC00CDB.9060109@lemburg.com> Message-ID: <15808.9056.873174.779158@gargle.gargle.HOWL> [Apologies for any dupes: I'm trying to repost this with one CC header to see if that fixes a problem Guido was seeing -BAW] >>>>> "MAL" == M writes: >> Not in the source distro, but it will end up in the RPMs and >> other binary distros. MAL> But that would violate the Sleepycat license; we are not MAL> shipping readline or a module which statically links against MAL> readline in the binary distros for the same reason. Again, when we talked to Sleepycat a year and a half ago, they said it would be ok to distribute the library with Python. IIRC, even a binary distro would be fine. But I think a simple email to them would clear the issue up. MAL> BTW, Sleepycat ships Berkley DB 4.1, wouldn't it be wise to MAL> update the wrapper module to that version ? Because there are problems: http://sourceforge.net/mailarchive/forum.php?thread_id=1206938&forum_id=4362 -Barry From martin@v.loewis.de Wed Oct 30 19:31:41 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 30 Oct 2002 20:31:41 +0100 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: <15807.59887.992769.142566@gargle.gargle.HOWL> References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> <200210301345.g9UDjxb21402@pcp02138704pcs.reston01.va.comcast.net> <15807.59887.992769.142566@gargle.gargle.HOWL> Message-ID: barry@python.org (Barry A. Warsaw) writes: > I'm not sure we'd want to distribute either the source or binary > though. They're fairly large. If we did, I'd recommend BerkeleyDB > 4.0.14, not the latest version, because bsddb3 doesn't yet build on > 4.1.24 because of some incompatible API changes, IIUC. libdb33.dll is 470k, which might be acceptable. I agree that distributing the source is pointless. > Martin, what do you plan to do about the pybsddb test suite? I would ignore it, or put it in non-dist (in case this indicates the end of the pybsddb project). > Maybe we include none of that, but simply "import" the code into > Python and re-sync whenever it's appropriate. That is my proposal. Initially, there would be no user-visible change, as it would operate in compatible mode. We could then incorporate more documentation as we go along. Regards, Martin From martin@v.loewis.de Wed Oct 30 19:52:25 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 30 Oct 2002 20:52:25 +0100 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: <200210301531.g9UFVW918144@odiug.zope.com> References: <200210300138.g9U1cse31882@pcp02138704pcs.reston01.va.comcast.net> <200210300512.g9U5CZK20380@pcp02138704pcs.reston01.va.comcast.net> <200210301531.g9UFVW918144@odiug.zope.com> Message-ID: Guido van Rossum writes: > It's similar to GPL's "copyleft". I think it's no different from what > we do with e.g. GNU readline, so I think it's okay. Redistributors of > Python in binary form will have to beware though. I wonder if we're > on thin ice with the RPMs (since we don't clarify any of this)? The RPMs should be fine: they include neither readline, nor bsddb. Instead, those libraries must be present on the target system. Regards, Martin From cpr@emsoftware.com Wed Oct 30 20:15:55 2002 From: cpr@emsoftware.com (Chris Ryland) Date: Wed, 30 Oct 2002 15:15:55 -0500 Subject: [Python-Dev] Fw: LAR Workshop Website Online, Message-ID: <12d401c28051$392b23b0$6701a8c0@EM2> Are you PythonLabs folks aware of this? It might be interesting &&|| relevant. -- Cheers! Chris Ryland Em Software, Inc. www.emsoftware.com > -----Original Message----- > From: Yahya H. Mirza [mailto:yahya_mirza@hotmail.com] > Sent: Wednesday, October 30, 2002 9:08 AM > To: Corin Day > Cc: J=FCrg Gutknecht; davidi@borland.com; Ralph Johnson; George Bosworth; Stan > Lippman; Yukihiro Matsumoto; eliot@parcplace.com; David Simmons; Gregory T. > Sullivan; David Stutz; Dan Sugalski; ian.piumarta@inria.fr; David Ungar; > Shawn Woods; Misha Dmitriev; Jim Miller (COM+); Tony Williams; Dan Fay > (RESEARCH); Mark Ryland; Andrew Palay; Mark Lewin; Don Syme; Greg O'Shea; > Pierre Sainoyant; Van Eden; Alan Borning; chambers@cs.washington.edu; Marc > E. Fiuczynski; David Notkin; zahorjan@cs.washington.edu; > jonal@cs.washington.edu; csk@cs.washington.edu; Brad Merrill; Brian Pepin; > Brian Foote > Subject: LAR Workshop Website Online, > > > I would like to invite you all to register for LAR02: > > Our workshop website is now online and it can be accessed either through the > main OOPSLA Workshop page or directly via: > > http://hosting.msugs.ch/auroraborealis/Workshops/LAR02/LAR02Home.htm > > > If you would please take the time to Register even if you are participating, > and provide some feedback, I would greatly appreciate it. > > Our workshop starts at 8:45 AM this Monday November 4 at the OOPSLA > Conference in the Seattle Convention Center in Rooms 613 - 614. > > Please feel free to invite whoever you think would be interested in > participating. > > Finally I would like to thank Cori Day for doing a wonderful job on the > website. > > Sincerely, > > Yahya > > Yahya H. Mirza > Aurora Borealis Software > 8502 166th Ave. NE > Redmond WA, 98052 > (425)-861-8147 From guido@python.org Wed Oct 30 20:44:38 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 30 Oct 2002 15:44:38 -0500 Subject: [Python-Dev] metaclass insanity In-Reply-To: Your message of "30 Oct 2002 18:17:26 GMT." <2my98faj0p.fsf@starship.python.net> References: <2md6psyowr.fsf@starship.python.net> <200210301644.g9UGiaZ18410@odiug.zope.com> <2my98faj0p.fsf@starship.python.net> Message-ID: <200210302044.g9UKicx22801@odiug.zope.com> > > The best thing to do would perhaps to make __mro__ assignable, but > > with a check that ensures the above constraint. I think I'd take a > > patch for that. > > Shouldn't be too hard. > > > I'd also take a patch for assignable __bases__. Note that there are > > constraints between __bases__ and __base__. > > Should assigning to __bases__ automatically tweak __mro__ and > __base__? Guess so. Yes. Note that changing __base__ should not be done lightly -- basically, the old and new base must be layout compatible, exactly like for assignment to __class__. > What would assigning to __base__ do in isolation? > Perhaps that shouldn't be writeable. Perhaps it could be writable when __bases__ is a 1-tuple. But it's fine if it's not writable. > > I'd also take a patch for assignable __name__. > > This is practically a one-liner, isn't it? Not hard, anyway. Probably. Can't remember why I didn't do it earlier. > And there was me wondering what I was going to do this evening. I don't have that problem -- a Zope customer problem was waiting for me today. :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 30 20:54:30 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 30 Oct 2002 15:54:30 -0500 Subject: [Python-Dev] Fw: LAR Workshop Website Online, In-Reply-To: Your message of "Wed, 30 Oct 2002 15:15:55 EST." <12d401c28051$392b23b0$6701a8c0@EM2> References: <12d401c28051$392b23b0$6701a8c0@EM2> Message-ID: <200210302054.g9UKsUn22895@odiug.zope.com> > Are you PythonLabs folks aware of this? It might be interesting &&|| > relevant. But I won't be able to make it. :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Wed Oct 30 14:19:11 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 30 Oct 2002 08:19:11 -0600 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: References: Message-ID: <15807.59999.240430.849098@montanaro.dyndns.org> Martin> I have changes in my sandbox to incorporate bsddb3 into Python Martin> proper.... Martin> Both test_bsddb and test_anydbm continue to work unmodified. Martin> Is it ok to commit these changes? I say go for it. Better now, when it's easily backed out than later. I use bsddb pretty frequently, and run Python CVS on my development machine, so I'll be able to wring it out a bit. Skip From martin@v.loewis.de Wed Oct 30 21:43:54 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 30 Oct 2002 22:43:54 +0100 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: <15807.59999.240430.849098@montanaro.dyndns.org> References: <15807.59999.240430.849098@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > I say go for it. Better now, when it's easily backed out than later. I use > bsddb pretty frequently, and run Python CVS on my development machine, so > I'll be able to wring it out a bit. Thanks for the encouragement. I really need PythonLab's permission here, though, since it will affect what the Windows distribution contains. If the political issues can be settled, working out the build process for Windows should be easy. Regards, Martin From guido@python.org Wed Oct 30 22:01:17 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 30 Oct 2002 17:01:17 -0500 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: Your message of "30 Oct 2002 22:43:54 +0100." References: <15807.59999.240430.849098@montanaro.dyndns.org> Message-ID: <200210302201.g9UM1Hm27151@odiug.zope.com> > Thanks for the encouragement. I really need PythonLab's permission > here, though, since it will affect what the Windows distribution > contains. > > If the political issues can be settled, working out the build process > for Windows should be easy. Barry will send an email to Sleepycat to ask exactly what the restrictions on binary redistribution are. There's also the issue of upgrading. Suppose someone has created a database using bsddb in Python 2.2. When they upgrade to Python 2.3, they'll get an exception when they open the database, because Sleepycat DB 4.x can't open BSD DB 1.85 databases. What do we tell them to do? --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Wed Oct 30 22:07:35 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 30 Oct 2002 17:07:35 -0500 Subject: [Python-Dev] Incorporating bsddb3 References: <15807.59999.240430.849098@montanaro.dyndns.org> Message-ID: <15808.22567.772352.109103@gargle.gargle.HOWL> >>>>> "MvL" == Martin v Loewis writes: MvL> If the political issues can be settled, working out the build MvL> process for Windows should be easy. I'm going to contact Sleepycat and start a dialog with them on the licensing and binary redist issues. I'll Cc Guido so we'll get a definitive answer on that, hopefully soon. -Barry From barry@python.org Wed Oct 30 22:10:58 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 30 Oct 2002 17:10:58 -0500 Subject: [Python-Dev] Incorporating bsddb3 References: <15807.59999.240430.849098@montanaro.dyndns.org> <200210302201.g9UM1Hm27151@odiug.zope.com> Message-ID: <15808.22770.456261.728316@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> There's also the issue of upgrading. Suppose someone has GvR> created a database using bsddb in Python 2.2. When they GvR> upgrade to Python 2.3, they'll get an exception when they GvR> open the database, because Sleepycat DB 4.x can't open BSD DB GvR> 1.85 databases. What do we tell them to do? Run this script: http://www.sleepycat.com/docs/utility/db_upgrade.html -Barry From skip@pobox.com Wed Oct 30 22:30:57 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 30 Oct 2002 16:30:57 -0600 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: <200210302201.g9UM1Hm27151@odiug.zope.com> References: <15807.59999.240430.849098@montanaro.dyndns.org> <200210302201.g9UM1Hm27151@odiug.zope.com> Message-ID: <15808.23969.429522.883750@montanaro.dyndns.org> Guido> When they upgrade to Python 2.3, they'll get an exception when Guido> they open the database, because Sleepycat DB 4.x can't open BSD Guido> DB 1.85 databases. What do we tell them to do? The same thing you've had to tell them in the past. Use the tools Sleepycat provides (db_dump, db_load, etc) to update databases from old file formats to new ones. Skip From martin@v.loewis.de Thu Oct 31 06:38:35 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 31 Oct 2002 07:38:35 +0100 Subject: [Python-Dev] Incorporating bsddb3 In-Reply-To: <200210302201.g9UM1Hm27151@odiug.zope.com> References: <15807.59999.240430.849098@montanaro.dyndns.org> <200210302201.g9UM1Hm27151@odiug.zope.com> Message-ID: Guido van Rossum writes: > There's also the issue of upgrading. Suppose someone has created a > database using bsddb in Python 2.2. When they upgrade to Python 2.3, > they'll get an exception when they open the database, because > Sleepycat DB 4.x can't open BSD DB 1.85 databases. What do we tell > them to do? Whether this happens somewhat depends on the system. On Linux, it won't happen, since people likely have been using Sleepycat DB, anyway, through db_185.h. If they really use DB 1.85 (i.e. on Windows), they need to use db_dump/db_load. Regards, Martin From barry@python.org Thu Oct 31 06:51:45 2002 From: barry@python.org (Barry A. Warsaw) Date: Thu, 31 Oct 2002 01:51:45 -0500 Subject: [Python-Dev] Incorporating bsddb3 References: <15807.59999.240430.849098@montanaro.dyndns.org> <200210302201.g9UM1Hm27151@odiug.zope.com> Message-ID: <15808.54017.738989.76521@gargle.gargle.HOWL> >>>>> "MvL" == Martin v Loewis writes: >> There's also the issue of upgrading. Suppose someone has >> created a database using bsddb in Python 2.2. When they >> upgrade to Python 2.3, they'll get an exception when they open >> the database, because Sleepycat DB 4.x can't open BSD DB 1.85 >> databases. What do we tell them to do? MvL> Whether this happens somewhat depends on the system. On MvL> Linux, it won't happen, since people likely have been using MvL> Sleepycat DB, anyway, through db_185.h. Also note that just because they used the 1.85 version of the API doesn't necessarily mean they've got a 1.85 version of the database file format. They could be using a newer version of the library in 1.85-compat mode. -Barry From mwh@python.net Thu Oct 31 10:34:57 2002 From: mwh@python.net (Michael Hudson) Date: 31 Oct 2002 10:34:57 +0000 Subject: [Python-Dev] metaclass insanity In-Reply-To: Guido van Rossum's message of "Wed, 30 Oct 2002 15:44:38 -0500" References: <2md6psyowr.fsf@starship.python.net> <200210301644.g9UGiaZ18410@odiug.zope.com> <2my98faj0p.fsf@starship.python.net> <200210302044.g9UKicx22801@odiug.zope.com> Message-ID: <2mznsu3nhq.fsf@starship.python.net> Guido van Rossum writes: > > > The best thing to do would perhaps to make __mro__ assignable, but > > > with a check that ensures the above constraint. I think I'd take a > > > patch for that. > > > > Shouldn't be too hard. > > > > > I'd also take a patch for assignable __bases__. Note that there are > > > constraints between __bases__ and __base__. > > > > Should assigning to __bases__ automatically tweak __mro__ and > > __base__? Guess so. > > Yes. Note that changing __base__ should not be done lightly -- > basically, the old and new base must be layout compatible, exactly > like for assignment to __class__. OK. I can crib code from type_set_class, I guess. Or one could just allow assignment to __bases__ when __base__ doesn't change? __base__ is object for the majority of new-style classes, isn't it? Brr. There's a lot I don't know about post 2.2 typeobject.c. > > What would assigning to __base__ do in isolation? > > Perhaps that shouldn't be writeable. > > Perhaps it could be writable when __bases__ is a 1-tuple. Don't see the point of that. > But it's fine if it's not writable. Easier :) > > > I'd also take a patch for assignable __name__. > > > > This is practically a one-liner, isn't it? Not hard, anyway. > > Probably. Can't remember why I didn't do it earlier. It's a bit more complicated than that. What's the deal wrt. dots in tp_name? Is there any way for a user defined class to end up called "something.somthing_else"? Oh, and while we're at it, here's a bogosity: >>> class C(object): ... pass ... >>> C.__module__ '__main__' >>> C.__module__ = 1 >>> C.__module__ Traceback (most recent call last): File "", line 1, in ? AttributeError: __module__ caused by lax testing in type_set_module. > > And there was me wondering what I was going to do this evening. > > I don't have that problem -- a Zope customer problem was waiting for > me today. :-( Well, I didn't get it finished either. Fiddly, this stuff. Maybe by tomorrow. Cheers, M. -- The Internet is full. Go away. -- http://www.disobey.com/devilshat/ds011101.htm From hbl@st-andrews.ac.uk Thu Oct 31 16:18:36 2002 From: hbl@st-andrews.ac.uk (Hamish Lawson) Date: Thu, 31 Oct 2002 16:18:36 +0000 Subject: [Python-Dev] Possible ideas from Strongtalk (static typing for Smalltalk)? Message-ID: <5.1.1.6.0.20021031160812.03bb5390@spey.st-andrews.ac.uk> I recently came across an announcement about the the Strongtalk system, which contains the first fully developed strong, static type system for Smalltalk. I wondered whether there might be useful ideas there for those looking into static typing for Python. http://www.cs.ucsb.edu/projects/strongtalk/pages/index.html Hamish Lawson From guido@python.org Thu Oct 31 18:58:00 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 31 Oct 2002 13:58:00 -0500 Subject: [Python-Dev] metaclass insanity In-Reply-To: Your message of "31 Oct 2002 10:34:57 GMT." <2mznsu3nhq.fsf@starship.python.net> References: <2md6psyowr.fsf@starship.python.net> <200210301644.g9UGiaZ18410@odiug.zope.com> <2my98faj0p.fsf@starship.python.net> <200210302044.g9UKicx22801@odiug.zope.com> <2mznsu3nhq.fsf@starship.python.net> Message-ID: <200210311858.g9VIw0509968@odiug.zope.com> > > > Should assigning to __bases__ automatically tweak __mro__ and > > > __base__? Guess so. > > > > Yes. Note that changing __base__ should not be done lightly -- > > basically, the old and new base must be layout compatible, exactly > > like for assignment to __class__. > > OK. I can crib code from type_set_class, I guess. Or one could just > allow assignment to __bases__ when __base__ doesn't change? __base__ > is object for the majority of new-style classes, isn't it? But if you derive from a builtin type (e.g. list or dict), __base__ will be that. > Brr. There's a lot I don't know about post 2.2 typeobject.c. Me too. :-) > > > What would assigning to __base__ do in isolation? > > > Perhaps that shouldn't be writeable. > > > > Perhaps it could be writable when __bases__ is a 1-tuple. > > Don't see the point of that. > > > But it's fine if it's not writable. > > Easier :) Agreed. > > > > I'd also take a patch for assignable __name__. > > > > > > This is practically a one-liner, isn't it? Not hard, anyway. > > > > Probably. Can't remember why I didn't do it earlier. > > It's a bit more complicated than that. > > What's the deal wrt. dots in tp_name? Is there any way for a user > defined class to end up called "something.somthing_else"? I hope not. The dots are for extensions living inside a package; everything before the last dot ends up as __module__. > Oh, and while we're at it, here's a bogosity: > > >>> class C(object): > ... pass > ... > >>> C.__module__ > '__main__' > >>> C.__module__ = 1 > >>> C.__module__ > Traceback (most recent call last): > File "", line 1, in ? > AttributeError: __module__ > > caused by lax testing in type_set_module. Oops. Can you fix it? Or are there complications? Seems to be broken in 2.2 too. > > > And there was me wondering what I was going to do this evening. > > > > I don't have that problem -- a Zope customer problem was waiting for > > me today. :-( > > Well, I didn't get it finished either. Fiddly, this stuff. Maybe by > tomorrow. Great! I'll be offline Friday through Monday -- going to a weekend conference. --Guido van Rossum (home page: http://www.python.org/~guido/) From niemeyer@conectiva.com Thu Oct 31 23:40:46 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 31 Oct 2002 20:40:46 -0300 Subject: [Python-Dev] Becoming a python contributor Message-ID: <20021031204046.A32673@ibook.distro.conectiva> Hello! After a long time of inactivity, I hope to be back to some python hacking. Before getting into any coding, I'd like to discuss with you some conceptual and boring stuff I have in my mind. In the past I have given small contributions to the python standard distribution. Unfortunately (for myself), I slowed down until I stopped contributing, even though I have a great affect by the interpreter. Now I realize that one of the reasons I've stopped contributing is because there's a large inertia in getting stuff reviewed. The reason why this is happening seems more aparent now that I was off for a while: there's a small core of very busy developers working on core/essential/hard stuff *and* in code reviewing as well. At the same time, I've seen Guido and others bothered a few times because of the lack of man power. So the question is: how do I, a developer which feels capable of helping in python's development, can get some of the tasks which take your time out of your hands? Or even, how is it possible to improve some part of the development process by giving people like me some instructions? Also, isn't it easy to point out what's wrong in a commit from someone who is following the development process for a while than taking the time to review its code in the sourceforge patch system? My feeling is that the Python development is currently overly centralized, and that you might be suffering from that now, by being unable to handover some of your tasks to someone else. I feel that everytime I send a patch, besides being contributing, I'm also overloading you with more stuff to review and comment and etc. Perhaps the fallback costs for some wrong commit is too high now (did I heard someone mentioning subversion?)?! Could someone please discuss that issues with me, and perhaps just kick me away saying that I'm crazy? :-) -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From niemeyer@conectiva.com Thu Oct 31 23:47:32 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 31 Oct 2002 20:47:32 -0300 Subject: [Python-Dev] Getting python-bz2 into 2.3 Message-ID: <20021031204732.B32673@ibook.distro.conectiva> Now that I'm relieved by sharing with you my feelings, ;-) what's the best path to get python-bz2 module into Python 2.3? Do you think it's something which should be in the core, or it'd be better to keep it as an external module? The code is currently maintained at http://python-bz2.sf.net, if someone wants to have a look at it. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ]