From janssen at parc.com Tue Mar 2 19:35:23 2010 From: janssen at parc.com (Bill Janssen) Date: Tue, 2 Mar 2010 10:35:23 PST Subject: [Python-ideas] the Quantity pattern Message-ID: <57533.1267554923@parc.com> I was looking at Martin Fowler's Quantity pattern earlier. http://martinfowler.com/ap2/quantity.html I remember writing this up as an idea for Fortran back in the early 80's, only to find a CACM paper from 1978 exploring the idea: "Incorporation of Units into Programming Languages", Karr & Loveman, May 1978. But it would still be a cool idea for Python. Perhaps it's already there and I haven't noticed? Bill From bwinton at latte.ca Tue Mar 2 20:14:41 2010 From: bwinton at latte.ca (Blake Winton) Date: Tue, 02 Mar 2010 14:14:41 -0500 Subject: [Python-ideas] the Quantity pattern In-Reply-To: <57533.1267554923@parc.com> References: <57533.1267554923@parc.com> Message-ID: <4B8D63A1.2070409@latte.ca> On 10-03-02 13:35 , Bill Janssen wrote: > I was looking at Martin Fowler's Quantity pattern earlier. > > http://martinfowler.com/ap2/quantity.html > > I remember writing this up as an idea for Fortran back in the early > 80's, only to find a CACM paper from 1978 exploring the idea: > "Incorporation of Units into Programming Languages", Karr& Loveman, May > 1978. > > But it would still be a cool idea for Python. Perhaps it's already > there and I haven't noticed? Were you thinking of something like http://www.arandonohue.com/hg/hgwebdir.cgi/units/file/bde33c01abd3/units/__init__.py#l1 ? Later, Blake. From robert.kern at gmail.com Tue Mar 2 20:17:06 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 02 Mar 2010 13:17:06 -0600 Subject: [Python-ideas] the Quantity pattern In-Reply-To: <57533.1267554923@parc.com> References: <57533.1267554923@parc.com> Message-ID: On 2010-03-02 12:35 PM, Bill Janssen wrote: > I was looking at Martin Fowler's Quantity pattern earlier. > > http://martinfowler.com/ap2/quantity.html > > I remember writing this up as an idea for Fortran back in the early > 80's, only to find a CACM paper from 1978 exploring the idea: > "Incorporation of Units into Programming Languages", Karr& Loveman, May > 1978. > > But it would still be a cool idea for Python. Perhaps it's already > there and I haven't noticed? Tons of implementations (in no particular order): http://pypi.python.org/pypi/quantities/ http://pypi.python.org/pypi/Unum/ http://pypi.python.org/pypi/magnitude/ http://pypi.python.org/pypi/units/ http://pypi.python.org/pypi/ScientificPython/ http://pypi.python.org/pypi/SciMath/ And quite a few more that are part of other packages or otherwise not on PyPI. It's ridiculously easy to write something that what people think are the common cases and so everyone does. It's a lot harder to write something that robustly handles what are actually common cases (absolute temperature scales, logarithmic scales, etc.). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From george.sakkis at gmail.com Tue Mar 2 20:37:29 2010 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 2 Mar 2010 20:37:29 +0100 Subject: [Python-ideas] the Quantity pattern In-Reply-To: References: <57533.1267554923@parc.com> Message-ID: <91ad5bf81003021137l2cbad439w21cd0f7e09a1a91d@mail.gmail.com> On Tue, Mar 2, 2010 at 8:17 PM, Robert Kern wrote: > On 2010-03-02 12:35 PM, Bill Janssen wrote: >> >> I was looking at Martin Fowler's Quantity pattern earlier. >> >> http://martinfowler.com/ap2/quantity.html >> >> I remember writing this up as an idea for Fortran back in the early >> 80's, only to find a CACM paper from 1978 exploring the idea: >> "Incorporation of Units into Programming Languages", Karr& ?Loveman, May >> 1978. >> >> But it would still be a cool idea for Python. ?Perhaps it's already >> there and I haven't noticed? > > Tons of implementations (in no particular order): > > http://pypi.python.org/pypi/quantities/ > http://pypi.python.org/pypi/Unum/ > http://pypi.python.org/pypi/magnitude/ > http://pypi.python.org/pypi/units/ > http://pypi.python.org/pypi/ScientificPython/ > http://pypi.python.org/pypi/SciMath/ > > And quite a few more that are part of other packages or otherwise not on > PyPI. It's ridiculously easy to write something that what people think are > the common cases and so everyone does. It's a lot harder to write something > that robustly handles what are actually common cases (absolute temperature > scales, logarithmic scales, etc.). One more: http://pypi.python.org/pypi/piquant/ I can't comment on its robustness and performance but as far as readability goes, Unum seems the best of the bunch. George From anfedorov at gmail.com Tue Mar 2 21:39:57 2010 From: anfedorov at gmail.com (Andrey Fedorov) Date: Tue, 2 Mar 2010 15:39:57 -0500 Subject: [Python-ideas] Matching multiple regex patterns simultaneously Message-ID: <7659cab31003021239x76ba71cdo4c691f256776f60@mail.gmail.com> So a couple of libraries (Django being the most popular that comes to mind) try to match a string against several regex expressions. I'm wondering if there exists a library to "merge" multiple compiled regex expressions into a single lookup. This could be exposed in a interface like: http://gist.github.com/319905 So for an example: rd = ReDict() rd['^foo$'] = 1 rd['^bar*$'] = 2 rd['^bar$'] = 3 assert rd['foo'] == [1] assert rd['barrrr'] == [2] assert rd['bar'] == [2,3] The naive implementation I link is obviously inefficient. What would be the easiest way to go about compiling a set of regex-es together, so that they can be matched against a string at the same time? Are there any standard libraries that do this I'm not aware of? Cheers, Andrey -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Mar 2 23:09:34 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 02 Mar 2010 23:09:34 +0100 Subject: [Python-ideas] Matching multiple regex patterns simultaneously In-Reply-To: <7659cab31003021239x76ba71cdo4c691f256776f60@mail.gmail.com> References: <7659cab31003021239x76ba71cdo4c691f256776f60@mail.gmail.com> Message-ID: <4B8D8C9E.5060201@egenix.com> Andrey Fedorov wrote: > So a couple of libraries (Django being the most popular that comes to mind) > try to match a string against several regex expressions. I'm wondering if > there exists a library to "merge" multiple compiled regex expressions into a > single lookup. This could be exposed in a interface like: > > http://gist.github.com/319905 > > > So for an example: > > rd = ReDict() > > rd['^foo$'] = 1 > rd['^bar*$'] = 2 > rd['^bar$'] = 3 > > assert rd['foo'] == [1] > assert rd['barrrr'] == [2] > assert rd['bar'] == [2,3] > > The naive implementation I link is obviously inefficient. What would be the > easiest way to go about compiling a set of regex-es together, so that they > can be matched against a string at the same time? Are there any standard > libraries that do this I'm not aware of? This is a rather difficult problem to solve in general. If you only need to search for a finite set of words, there are few good algorithms for this: http://en.wikipedia.org/wiki/Aho-Corasick_algorithm * used in Unix fgrep * Python implementation: http://hkn.eecs.berkeley.edu/~dyoo/python/ahocorasick/ http://en.wikipedia.org/wiki/Rabin-Karp_string_search_algorithm#Rabin.E2.80.93Karp_and_multiple_pattern_search * uses hashing -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 02 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From grosser.meister.morti at gmx.net Tue Mar 2 23:32:42 2010 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Tue, 02 Mar 2010 23:32:42 +0100 Subject: [Python-ideas] Matching multiple regex patterns simultaneously In-Reply-To: <7659cab31003021239x76ba71cdo4c691f256776f60@mail.gmail.com> References: <7659cab31003021239x76ba71cdo4c691f256776f60@mail.gmail.com> Message-ID: <4B8D920A.1010309@gmx.net> On 03/02/2010 09:39 PM, Andrey Fedorov wrote: > So a couple of libraries (Django being the most popular that comes to > mind) try to match a string against several regex expressions. I'm > wondering if there exists a library to "merge" multiple compiled regex > expressions into a single lookup. This could be exposed in a interface like: > > http://gist.github.com/319905 > > > So for an example: > > rd = ReDict() > > rd['^foo$'] = 1 > rd['^bar*$'] = 2 > rd['^bar$'] = 3 > > assert rd['foo'] == [1] > assert rd['barrrr'] == [2] > assert rd['bar'] == [2,3] > > The naive implementation I link is obviously inefficient. What would be > the easiest way to go about compiling a set of regex-es together, so > that they can be matched against a string at the same time? Are there > any standard libraries that do this I'm not aware of? > > Cheers, > Andrey > You can do something like this: r=re.compile('(?P^foo$)|(?P(?P^bar)r*$)') >>> r.match('barrrr').groupdict() {'a': None, 'c': 'bar', 'b': 'barrrr'} >>> r.match('bar').groupdict() {'a': None, 'c': 'bar', 'b': 'bar'} >>> r.match('foo').groups() ('foo', None, None) Ok, it's not 100% the same (it does not match 'ba'), but I think this should cover most cases where you want something like this. Hmm, well. You should resolve it to a form where there are no overlappings in the subexpressions: (?P^foo$)|(?P^ba$)|(?P^bar$)|(?P^bar+$) -panzi From mwm-keyword-python.b4bdba at mired.org Wed Mar 3 01:17:43 2010 From: mwm-keyword-python.b4bdba at mired.org (Mike Meyer) Date: Tue, 2 Mar 2010 19:17:43 -0500 Subject: [Python-ideas] the Quantity pattern In-Reply-To: <91ad5bf81003021137l2cbad439w21cd0f7e09a1a91d@mail.gmail.com> References: <57533.1267554923@parc.com> <91ad5bf81003021137l2cbad439w21cd0f7e09a1a91d@mail.gmail.com> Message-ID: <20100302191743.7d7b6f77@bhuda.mired.org> On Tue, 2 Mar 2010 20:37:29 +0100 George Sakkis wrote: > On Tue, Mar 2, 2010 at 8:17 PM, Robert Kern wrote: > > On 2010-03-02 12:35 PM, Bill Janssen wrote: > >> > >> I was looking at Martin Fowler's Quantity pattern earlier. > >> > >> http://martinfowler.com/ap2/quantity.html > >> > >> I remember writing this up as an idea for Fortran back in the early > >> 80's, only to find a CACM paper from 1978 exploring the idea: > >> "Incorporation of Units into Programming Languages", Karr& ?Loveman, May > >> 1978. > >> > >> But it would still be a cool idea for Python. ?Perhaps it's already > >> there and I haven't noticed? > > > > Tons of implementations (in no particular order): > > > > http://pypi.python.org/pypi/quantities/ > > http://pypi.python.org/pypi/Unum/ > > http://pypi.python.org/pypi/magnitude/ > > http://pypi.python.org/pypi/units/ > > http://pypi.python.org/pypi/ScientificPython/ > > http://pypi.python.org/pypi/SciMath/ > > > > And quite a few more that are part of other packages or otherwise not on > > PyPI. It's ridiculously easy to write something that what people think are > > the common cases and so everyone does. It's a lot harder to write something > > that robustly handles what are actually common cases (absolute temperature > > scales, logarithmic scales, etc.). > > One more: http://pypi.python.org/pypi/piquant/ > > I can't comment on its robustness and performance but as far as > readability goes, Unum seems the best of the bunch. Hmm. How about accessing the Frink (http://futureboy.homeip.net/frinkdocs/) types from Jython? http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From robert.kern at gmail.com Wed Mar 3 01:42:26 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 02 Mar 2010 18:42:26 -0600 Subject: [Python-ideas] the Quantity pattern In-Reply-To: <20100302191743.7d7b6f77@bhuda.mired.org> References: <57533.1267554923@parc.com> <91ad5bf81003021137l2cbad439w21cd0f7e09a1a91d@mail.gmail.com> <20100302191743.7d7b6f77@bhuda.mired.org> Message-ID: On 2010-03-02 18:17 PM, Mike Meyer wrote: > On Tue, 2 Mar 2010 20:37:29 +0100 > George Sakkis wrote: > >> On Tue, Mar 2, 2010 at 8:17 PM, Robert Kern wrote: >>> On 2010-03-02 12:35 PM, Bill Janssen wrote: >>>> >>>> I was looking at Martin Fowler's Quantity pattern earlier. >>>> >>>> http://martinfowler.com/ap2/quantity.html >>>> >>>> I remember writing this up as an idea for Fortran back in the early >>>> 80's, only to find a CACM paper from 1978 exploring the idea: >>>> "Incorporation of Units into Programming Languages", Karr& Loveman, May >>>> 1978. >>>> >>>> But it would still be a cool idea for Python. Perhaps it's already >>>> there and I haven't noticed? >>> >>> Tons of implementations (in no particular order): >>> >>> http://pypi.python.org/pypi/quantities/ >>> http://pypi.python.org/pypi/Unum/ >>> http://pypi.python.org/pypi/magnitude/ >>> http://pypi.python.org/pypi/units/ >>> http://pypi.python.org/pypi/ScientificPython/ >>> http://pypi.python.org/pypi/SciMath/ >>> >>> And quite a few more that are part of other packages or otherwise not on >>> PyPI. It's ridiculously easy to write something that what people think are >>> the common cases and so everyone does. It's a lot harder to write something >>> that robustly handles what are actually common cases (absolute temperature >>> scales, logarithmic scales, etc.). >> >> One more: http://pypi.python.org/pypi/piquant/ >> >> I can't comment on its robustness and performance but as far as >> readability goes, Unum seems the best of the bunch. > > Hmm. How about accessing the Frink > (http://futureboy.homeip.net/frinkdocs/) types from Jython? You might be able to use it to do the unit conversion calculations, but the Java interface to Frink does not expose any types following the Quantity pattern. It just provides a way to evaluate strings in the Frink interpreter: http://futureboy.homeip.net/frinkdocs/integrate/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dsdale24 at gmail.com Wed Mar 3 03:02:31 2010 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 2 Mar 2010 21:02:31 -0500 Subject: [Python-ideas] the Quantity pattern In-Reply-To: <91ad5bf81003021137l2cbad439w21cd0f7e09a1a91d@mail.gmail.com> References: <57533.1267554923@parc.com> <91ad5bf81003021137l2cbad439w21cd0f7e09a1a91d@mail.gmail.com> Message-ID: On Tue, Mar 2, 2010 at 2:37 PM, George Sakkis wrote: > On Tue, Mar 2, 2010 at 8:17 PM, Robert Kern wrote: >> On 2010-03-02 12:35 PM, Bill Janssen wrote: >>> >>> I was looking at Martin Fowler's Quantity pattern earlier. >>> >>> http://martinfowler.com/ap2/quantity.html >>> >>> I remember writing this up as an idea for Fortran back in the early >>> 80's, only to find a CACM paper from 1978 exploring the idea: >>> "Incorporation of Units into Programming Languages", Karr& ?Loveman, May >>> 1978. >>> >>> But it would still be a cool idea for Python. ?Perhaps it's already >>> there and I haven't noticed? >> >> Tons of implementations (in no particular order): >> >> http://pypi.python.org/pypi/quantities/ >> http://pypi.python.org/pypi/Unum/ >> http://pypi.python.org/pypi/magnitude/ >> http://pypi.python.org/pypi/units/ >> http://pypi.python.org/pypi/ScientificPython/ >> http://pypi.python.org/pypi/SciMath/ >> >> And quite a few more that are part of other packages or otherwise not on >> PyPI. It's ridiculously easy to write something that what people think are >> the common cases and so everyone does. It's a lot harder to write something >> that robustly handles what are actually common cases (absolute temperature >> scales, logarithmic scales, etc.). > > One more: http://pypi.python.org/pypi/piquant/ > > I can't comment on its robustness and performance but as far as > readability goes, Unum seems the best of the bunch. I am the developer of the Quantities package. The tutorial at http://packages.python.org/quantities/user/tutorial.html recommends importing the units and constants into a namespace, but aside from that, the syntax seems very similar to Unum. However, quantities depends on numpy. Darren From dsdale24 at gmail.com Wed Mar 3 03:14:57 2010 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 2 Mar 2010 21:14:57 -0500 Subject: [Python-ideas] the Quantity pattern In-Reply-To: References: <57533.1267554923@parc.com> Message-ID: On Tue, Mar 2, 2010 at 2:17 PM, Robert Kern wrote: > On 2010-03-02 12:35 PM, Bill Janssen wrote: >> >> I was looking at Martin Fowler's Quantity pattern earlier. >> >> http://martinfowler.com/ap2/quantity.html >> >> I remember writing this up as an idea for Fortran back in the early >> 80's, only to find a CACM paper from 1978 exploring the idea: >> "Incorporation of Units into Programming Languages", Karr& ?Loveman, May >> 1978. >> >> But it would still be a cool idea for Python. ?Perhaps it's already >> there and I haven't noticed? > > Tons of implementations (in no particular order): > > http://pypi.python.org/pypi/quantities/ > http://pypi.python.org/pypi/Unum/ > http://pypi.python.org/pypi/magnitude/ > http://pypi.python.org/pypi/units/ > http://pypi.python.org/pypi/ScientificPython/ > http://pypi.python.org/pypi/SciMath/ > > And quite a few more that are part of other packages or otherwise not on > PyPI. It's ridiculously easy to write something that what people think are > the common cases and so everyone does. It's a lot harder to write something > that robustly handles what are actually common cases (absolute temperature > scales, logarithmic scales, etc.). I prefer to think of this as two separate issues. One issue is a Quantity pattern for dealing with values that have magnitude and dimensionality, and the other is coordinate systems (requiring a point of reference, like temperature scales). Darren From robert.kern at gmail.com Wed Mar 3 04:35:00 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 02 Mar 2010 21:35:00 -0600 Subject: [Python-ideas] the Quantity pattern In-Reply-To: References: <57533.1267554923@parc.com> Message-ID: On 2010-03-02 20:14 , Darren Dale wrote: > On Tue, Mar 2, 2010 at 2:17 PM, Robert Kern wrote: >> On 2010-03-02 12:35 PM, Bill Janssen wrote: >>> >>> I was looking at Martin Fowler's Quantity pattern earlier. >>> >>> http://martinfowler.com/ap2/quantity.html >>> >>> I remember writing this up as an idea for Fortran back in the early >>> 80's, only to find a CACM paper from 1978 exploring the idea: >>> "Incorporation of Units into Programming Languages", Karr& Loveman, May >>> 1978. >>> >>> But it would still be a cool idea for Python. Perhaps it's already >>> there and I haven't noticed? >> >> Tons of implementations (in no particular order): >> >> http://pypi.python.org/pypi/quantities/ >> http://pypi.python.org/pypi/Unum/ >> http://pypi.python.org/pypi/magnitude/ >> http://pypi.python.org/pypi/units/ >> http://pypi.python.org/pypi/ScientificPython/ >> http://pypi.python.org/pypi/SciMath/ >> >> And quite a few more that are part of other packages or otherwise not on >> PyPI. It's ridiculously easy to write something that what people think are >> the common cases and so everyone does. It's a lot harder to write something >> that robustly handles what are actually common cases (absolute temperature >> scales, logarithmic scales, etc.). > > I prefer to think of this as two separate issues. One issue is a > Quantity pattern for dealing with values that have magnitude and > dimensionality, and the other is coordinate systems (requiring a point > of reference, like temperature scales). Theoretically and implementation-wise, absolutely. However, users want to convert Fahrenheit to Celsius with the same tool they use to convert meters to feet. To them, it's the same problem. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan_ml at behnel.de Wed Mar 3 09:37:27 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 03 Mar 2010 09:37:27 +0100 Subject: [Python-ideas] Matching multiple regex patterns simultaneously In-Reply-To: <4B8D8C9E.5060201@egenix.com> References: <7659cab31003021239x76ba71cdo4c691f256776f60@mail.gmail.com> <4B8D8C9E.5060201@egenix.com> Message-ID: M.-A. Lemburg, 02.03.2010 23:09: > If you only need to search for a finite set of words, there are few > good algorithms for this: > > http://en.wikipedia.org/wiki/Aho-Corasick_algorithm > * used in Unix fgrep > * Python implementation: > http://hkn.eecs.berkeley.edu/~dyoo/python/ahocorasick/ > > http://en.wikipedia.org/wiki/Rabin-Karp_string_search_algorithm#Rabin.E2.80.93Karp_and_multiple_pattern_search > * uses hashing ... and acora: http://pypi.python.org/pypi/acora Stefan From mikegraham at gmail.com Wed Mar 3 20:48:18 2010 From: mikegraham at gmail.com (Mike Graham) Date: Wed, 3 Mar 2010 13:48:18 -0600 Subject: [Python-ideas] Function caller in operator module Message-ID: <3f588a651003031148gc35ca37o7bb609c02f2e61fa@mail.gmail.com> Most operations that are available using operators in Python are provided by the operator module, but calling functions is noticeably absent is calling functions. A user can slightly abuse operator.methodcaller('__call__', ...) to perform this operation, but that is far from ideal. Should operator grow a new function caller, such that operator.caller(*args, **kwargs)(f) returns f(*args, **kwargs)? I have never personally needed this exact operation; it would be somewhat odd but not altogether unthinkable to sort a list of callables by their return values. The closes thing to this I have seen in the wild is map(apply, fs), which is limited to the no-arguments case. It is also possibly uglier than [f() for f in fs]. In any event, it's not even an option in Python 3.x. From tjreedy at udel.edu Wed Mar 3 22:21:16 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 03 Mar 2010 16:21:16 -0500 Subject: [Python-ideas] Function caller in operator module In-Reply-To: <3f588a651003031148gc35ca37o7bb609c02f2e61fa@mail.gmail.com> References: <3f588a651003031148gc35ca37o7bb609c02f2e61fa@mail.gmail.com> Message-ID: On 3/3/2010 2:48 PM, Mike Graham wrote: > Most operations that are available using operators in Python are > provided by the operator module, but calling functions is noticeably > absent is calling functions. A user can slightly abuse > operator.methodcaller('__call__', ...) to perform this operation, but > that is far from ideal. > > Should operator grow a new function caller, such that > operator.caller(*args, **kwargs)(f) returns f(*args, **kwargs)? Use case? > I have never personally needed this exact operation; it would be > somewhat odd but not altogether unthinkable to sort a list of > callables by their return values. The closes thing to this I have seen > in the wild is map(apply, fs), which is limited to the no-arguments > case. It is also possibly uglier than [f() for f in fs]. In any event, > it's not even an option in Python 3.x. And not needed, as you show. tjr From mal at egenix.com Thu Mar 4 01:21:07 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 04 Mar 2010 01:21:07 +0100 Subject: [Python-ideas] Matching multiple regex patterns simultaneously In-Reply-To: References: <7659cab31003021239x76ba71cdo4c691f256776f60@mail.gmail.com> <4B8D8C9E.5060201@egenix.com> Message-ID: <4B8EFCF3.80109@egenix.com> Stefan Behnel wrote: > M.-A. Lemburg, 02.03.2010 23:09: >> If you only need to search for a finite set of words, there are few >> good algorithms for this: >> >> http://en.wikipedia.org/wiki/Aho-Corasick_algorithm >> * used in Unix fgrep >> * Python implementation: >> http://hkn.eecs.berkeley.edu/~dyoo/python/ahocorasick/ >> >> http://en.wikipedia.org/wiki/Rabin-Karp_string_search_algorithm#Rabin.E2.80.93Karp_and_multiple_pattern_search >> >> * uses hashing > > ... and acora: > > http://pypi.python.org/pypi/acora Nice :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 04 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From denis.spir at gmail.com Thu Mar 4 11:35:53 2010 From: denis.spir at gmail.com (spir) Date: Thu, 4 Mar 2010 11:35:53 +0100 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() Message-ID: <20100304113553.514db067@o> Hello, (1) I do not understand an iterable type's __iter__() method to be compulsary. Actually, each time I have defined one, I had to write: def __iter__(self): return self So, I guess that if python does not find __iter__(), but the object defines next(), then by default the said object could be used as its own iterator. This is what I understand by "iterable" and next() is the required method for it. Or even better: only if the object does not define next(), then python falls back to looking for __iter__(). Is there any obstacle for this I cannot see? Side-question: In which cases is it necessary to define the iterator as a separate object? (2) But: for any reason next() is not spelled as a "magic" method. If this method becomes the distinctive method of iterables, then it should be called __next__() for consistency. Side-question: Why is it called next(), as it is a magic method for iterators already? (3) What I miss actually for iterables (which are their own iterator) is a kind of __reset__(). In some cases, it is only needed to allow a new iteration from start. But it may even be needed to set some startup data the first time. __reset__() would thus be called once before the first call to next(). (Sure, "reset" may not be the best term. Maybe "begin" or "startup"? The sense is: "Prepare to yield the first item!") In absence of such a startup mechanism, I end up using __call__ instead: Example: class Powers(object): def __init__(self, exponent): self.exponent = exponent def next(self): n = self.n + 1 if (self.max is not None) and (n > self.max): raise StopIteration self.n = n return n*n def __call__(self, min=1,max=None): self.n = min-1 self.max = max return self #.__iter__() def __iter__(self): return self tripleCubes = [] cubes = Powers(3) for sq in cubes(7,17): if sq%3 == 0: tripleCubes.append(sq) print tripleCubes # ==> [81, 144, 225] __iter__() could be used directly if it would allow "free" args in addition to self (in this case: def __iter__(self, min=0,max=None). This beeing impossible, an aditional method seems to be needed. To sum up, I would enjoy beeing able to write Powers using a scheme like: class Powers(object): def __init__(self, exponent): self.exponent = exponent def __reset__(self, min=1,max=None): self.n = min-1 self.max = max def __next__(self): n = self.n + 1 if (self.max is not None) and (n > self.max): raise StopIteration self.n = n return n*n This matches the overall idea of iterable for me. Denis -- ________________________________ la vita e estrany spir.wikidot.com From masklinn at masklinn.net Thu Mar 4 12:07:48 2010 From: masklinn at masklinn.net (Masklinn) Date: Thu, 4 Mar 2010 12:07:48 +0100 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() In-Reply-To: <20100304113553.514db067@o> References: <20100304113553.514db067@o> Message-ID: On 4 Mar 2010, at 11:35 , spir wrote: > > (3) What I miss actually for iterables (which are their own iterator) is a kind of __reset__(). In some cases, it is only needed to allow a new iteration from start. But it may even be needed to set some startup data the first time. __reset__() would thus be called once before the first call to next(). (Sure, "reset" may not be the best term. Maybe "begin" or "startup"? The sense is: "Prepare to yield the first item!") The use case you propose (and demonstrate in your example) shows that you're creating an iterating view over your container. Which doesn't seem the role of __iter__ at all as far as I understand, so you should indeed use either __call__ or a separate method call. Slice, for instance. In fact, in your case I think supporting slicing/islicing would make far more sense than the solution you've elected): tripleCubes = [] cubes = Powers(3) for sq in cubes[6, 17]: if sq%3 == 0: tripleCubes.append(sq) You could also compose your iterable with existing tools such as those in ``itertools``, or create your own composable iterable transformers/manipulators (I strongly recommend David M. Beazley's presentations on generators for that [1][2]. With those, removing __call__ from your Power[3] class and setting `self.n = 0` (and `self.max = None`) in the constructor: cubes = Powers(3) tripleCubes = filter(lambda sq: sq%3 == 0, (islice(cubes, 6, 17))) Or (to avoid the pretty ugly lambda and use a listcomp): cubes = Powers(3) tripleCubes = [sq for sq in islice(cubes, 6, 16) if sq%3 == 0] [1] Generator Tricks for Systems Programmers http://www.dabeaz.com/generators/ [2] A Curious Course on Coroutines and Concurrency http://www.dabeaz.com/coroutines/ [3] I hope and believe you wouldn't actually write such code outside of an example as there are much better ways to achieve the same thing in Python From pyideas at rebertia.com Thu Mar 4 12:45:11 2010 From: pyideas at rebertia.com (Chris Rebert) Date: Thu, 4 Mar 2010 03:45:11 -0800 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() In-Reply-To: <20100304113553.514db067@o> References: <20100304113553.514db067@o> Message-ID: <50697b2c1003040345o24713191w54902939c326ab99@mail.gmail.com> On Thu, Mar 4, 2010 at 2:35 AM, spir wrote: > Hello, > > (1) I do not understand an iterable type's __iter__() method to be compulsary. Actually, each time I have defined one, I had to write: > ? ?def __iter__(self): > ? ? ? ?return self > So, I guess that if python does not find __iter__(), but the object defines next(), then by default the said object could be used as its own iterator. This is what I understand by "iterable" and next() is the required method for it. > Or even better: only if the object does not define next(), then python falls back to looking for __iter__(). Is there any obstacle for this I cannot see? Not that I can think of; Python just happened to make a different design decision than you, one that simplifies (and likely speeds up) the interpreter at the minor cost of having to write a trivial "return self" __iter__() in some cases: the interpreter can just blindly call __iter__() as opposed to (as you suggest) doing more sophisticated checking for a next() method and only then falling back to __iter__(). > Side-question: In which cases is it necessary to define the iterator as a separate object? Whenever you want to use multiple iterators over the same object simultaneously (you can usually equally use a generator instead of an object, but the iterator is separate from the iterate-ee in either case). For example, if lists were their own iterators, the following code: for item1 in some_list: for item2 in some_list: print item1, item2 rather than outputting the cross-product of the items in the list, would presumably instead output the first element paired with every other element in the list, which is not what was intended. > (2) But: for any reason next() is not spelled as a "magic" method. If this method becomes the distinctive method of iterables, then it should be called __next__() for consistency. Guido's time machine strikes again! This is fixed in Python 3.x: http://www.python.org/dev/peps/pep-3114/ > (3) What I miss actually for iterables (which are their own iterator) is a kind of __reset__(). In some cases, it is only needed to allow a new iteration from start. But it may even be needed to set some startup data the first time. __reset__() would thus be called once before the first call to next(). (a) __reset__() shouldn't be part of the iterator protocol since it's not applicable for all iterators, only some. (b) You can just write a generator and put the setup code before the initial "yield" to much the same effect. Cheers, Chris -- Maven & would-be designer of languages http://blog.rebertia.com From denis.spir at gmail.com Thu Mar 4 12:53:27 2010 From: denis.spir at gmail.com (spir) Date: Thu, 4 Mar 2010 12:53:27 +0100 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() In-Reply-To: References: <20100304113553.514db067@o> Message-ID: <20100304125327.436b1801@o> On Thu, 4 Mar 2010 12:07:48 +0100 Masklinn wrote: Thanks for your reply: I find it helpful; will have a look at the pointed references in a short while. > In fact, in your case I think supporting slicing/islicing would make far more sense than the solution you've elected) Actually my Powers type should be "sliceable". > tripleCubes = [sq for sq in islice(cubes, 6, 16) > if sq%3 == 0] Right, but this pattern only replaces cubes(6, 16) by islice(cubes, 6, 16): tripleCubes = [sq for sq in cubes(6, 16) if sq%3 == 0] Or do I overlook a relevant point? I guess the proper (and pythonic?) solution would be to implement __getitem__ so as to be able to write: tripleCubes = [sq for sq in cubes[6:17] if sq%3 == 0] Does the following match your idea? def __getitem__(self, ranj): self.n , self.max = ranj.start-1 , ranj.stop-1 return self Denis -- ________________________________ la vita e estrany spir.wikidot.com From masklinn at masklinn.net Thu Mar 4 13:41:14 2010 From: masklinn at masklinn.net (Masklinn) Date: Thu, 4 Mar 2010 13:41:14 +0100 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() In-Reply-To: <20100304125327.436b1801@o> References: <20100304113553.514db067@o> <20100304125327.436b1801@o> Message-ID: On 4 Mar 2010, at 12:53 , spir wrote: > > Right, but this pattern only replaces cubes(6, 16) by islice(cubes, 6, 16) Yes, the point is that you don't *need* cubes to be callable to do what you want. Because Python already provides the tools to do it in its stdlib. So you have no reason to concern yourself with that. > Does the following match your idea? > def __getitem__(self, ranj): > self.n , self.max = ranj.start-1 , ranj.stop-1 > return self I'd return a new item with the relevant attributes set, not an modified set (generally, I'm not fond of mutable object but that might just be me) From merwok at netwok.org Thu Mar 4 13:37:56 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 04 Mar 2010 13:37:56 +0100 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() In-Reply-To: <20100304113553.514db067@o> References: <20100304113553.514db067@o> Message-ID: <4B8FA9A4.2090907@netwok.org> Hello list Some complementary information to Chris Rebert?s message. > (1) I do not understand an iterable type's __iter__() method to be > compulsary. [...] The iterable and the iterator protocols are two different things. Every iterator is iterable, but not every iterable object is an iterator (see ). There are situations where it makes sense to use a different class as iterator (can?t find examples right now, hope other people will chime in), and a lot of situations where it?s fine to implement both protocols on the same class. > Actually, each time I have defined one, I had to write: > def __iter__(self): > return self Instead of returning self in __iter__ and then defining a function in next, i.e. returning values and raising StopIteration, you can define __iter__ as a generator (i.e. yield values instead of returning them) and not need to write next. Helpful documentation here: and > (3) What I miss actually for iterables (which are their own iterator) > is a kind of __reset__(). In some cases, it is only needed to allow a > new iteration from start.[...] Didn?t really understand your use case here, but perhaps the extension of the yield statement done in version 2.5 can help you here: Hope this helps. Regards From ncoghlan at gmail.com Thu Mar 4 14:23:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 04 Mar 2010 23:23:34 +1000 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() In-Reply-To: <20100304113553.514db067@o> References: <20100304113553.514db067@o> Message-ID: <4B8FB456.5050309@gmail.com> spir wrote: > Hello, > > (1) I do not understand an iterable type's __iter__() method to be > compulsary. Actually, each time I have defined one, I had to write: > def __iter__(self): return self So, I guess that if python does not > find __iter__(), but the object defines next(), then by default the > said object could be used as its own iterator. This is what I > understand by "iterable" and next() is the required method for it. Or > even better: only if the object does not define next(), then python > falls back to looking for __iter__(). Is there any obstacle for this > I cannot see? Side-question: In which cases is it necessary to define > the iterator as a separate object? Almost all containers should use a separate object for their iterators. Note that this is already the case for all of Python's standard container types. The reason relates to the __reset__ suggestion you describe later in your message: How do I reset an iterator over a list? Easy, just call iter() again - it will give me a fresh iterator that starts at the beginning without affecting the list or my original iterator. By producing a fresh object for each invocation of __iter__ the state of the iterators is decoupled from the state of the underlying object which is generally a good thing from a program design point of view. (See Eric's suggestion regarding the use of generators as __iter__ methods to easily achieve this behaviour) Objects with significant state that are also their own iterators are actually quite rare. File objects certainly qualify (since they base their iteration off the file object's file pointer), but I can't think of any others off the top of my head. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From brett at python.org Thu Mar 4 21:25:19 2010 From: brett at python.org (Brett Cannon) Date: Thu, 4 Mar 2010 12:25:19 -0800 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() In-Reply-To: <4B8FB456.5050309@gmail.com> References: <20100304113553.514db067@o> <4B8FB456.5050309@gmail.com> Message-ID: On Thu, Mar 4, 2010 at 05:23, Nick Coghlan wrote: > spir wrote: > > Hello, > > > > (1) I do not understand an iterable type's __iter__() method to be > > compulsary. Actually, each time I have defined one, I had to write: > > def __iter__(self): return self So, I guess that if python does not > > find __iter__(), but the object defines next(), then by default the > > said object could be used as its own iterator. This is what I > > understand by "iterable" and next() is the required method for it. Or > > even better: only if the object does not define next(), then python > > falls back to looking for __iter__(). Is there any obstacle for this > > I cannot see? Side-question: In which cases is it necessary to define > > the iterator as a separate object? > > Almost all containers should use a separate object for their iterators. > Note that this is already the case for all of Python's standard > container types. > There is also the issue of backwards-compatibility when iterators were introduced. Just because someone decided to have a method named next() when iterators were introduced does not mean they intended for it to be viewed as a sequence. Requiring an iterable to define __iter__() took care of the ambiguity. -Brett > > The reason relates to the __reset__ suggestion you describe later in > your message: How do I reset an iterator over a list? Easy, just call > iter() again - it will give me a fresh iterator that starts at the > beginning without affecting the list or my original iterator. By > producing a fresh object for each invocation of __iter__ the state of > the iterators is decoupled from the state of the underlying object which > is generally a good thing from a program design point of view. > > (See Eric's suggestion regarding the use of generators as __iter__ > methods to easily achieve this behaviour) > > Objects with significant state that are also their own iterators are > actually quite rare. File objects certainly qualify (since they base > their iteration off the file object's file pointer), but I can't think > of any others off the top of my head. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Mar 4 23:48:06 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 05 Mar 2010 11:48:06 +1300 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() In-Reply-To: <20100304113553.514db067@o> References: <20100304113553.514db067@o> Message-ID: <4B9038A6.2060909@canterbury.ac.nz> spir wrote: > (2) But: for any reason next() is not spelled as a "magic" method. If > this method becomes the distinctive method of iterables, then it > should be called __next__() for consistency. I gather that it is indeed called __next__() in Py3, and there is a new builtin function next() for invoking it. -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 5 00:07:55 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 05 Mar 2010 12:07:55 +1300 Subject: [Python-ideas] Matching multiple regex patterns simultaneously In-Reply-To: <7659cab31003021239x76ba71cdo4c691f256776f60@mail.gmail.com> References: <7659cab31003021239x76ba71cdo4c691f256776f60@mail.gmail.com> Message-ID: <4B903D4B.1050209@canterbury.ac.nz> Andrey Fedorov wrote: > So a couple of libraries (Django being the most popular that comes to mind) > try to match a string against several regex expressions. I'm wondering if > there exists a library to "merge" multiple compiled regex expressions into a > single lookup. The only thing I'm aware of at the moment is my own Plex library, but it's currently implemented in pure Python, so it's not as efficient as the re module with an extension like this would be. I have a half-finished project on the back burner to reimplement the core of Plex using Pyrex, but it's been languishing for so long that the burner has probably gone out by now. Maybe I should relight it... -- Greg From g.brandl at gmx.net Thu Mar 4 22:43:53 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 04 Mar 2010 22:43:53 +0100 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() In-Reply-To: <20100304113553.514db067@o> References: <20100304113553.514db067@o> Message-ID: Am 04.03.2010 11:35, schrieb spir: > Hello, > > (1) I do not understand an iterable type's __iter__() method to be > compulsary. Actually, each time I have defined one, I had to write: def > __iter__(self): return self So, I guess that if python does not find > __iter__(), but the object defines next(), then by default the said object > could be used as its own iterator. This is what I understand by "iterable" > and next() is the required method for it. Or even better: only if the object > does not define next(), then python falls back to looking for __iter__(). Is > there any obstacle for this I cannot see? Side-question: In which cases is it > necessary to define the iterator as a separate object? I would say that in most cases it makes sense to have the iterator be a separate object. However, when writing an __iter__() in Python, you almost always can make it a generator, which already does this for you -- each call to __iter__() will return a new generator. > (2) But: for any reason next() is not spelled as a "magic" method. If this > method becomes the distinctive method of iterables, then it should be called > __next__() for consistency. Side-question: Why is it called next(), as it is > a magic method for iterators already? Because it is supposed to be called directly. __iter__() isn't. (This changes with Python 3, where you have next() as a builtin.) As such, this is the same question as "why is it called readline(), not __readline__()." readline(), just like next(), is a method defined by a protocol (file vs iterator). Georg From solipsis at pitrou.net Fri Mar 5 00:54:05 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 4 Mar 2010 18:54:05 -0500 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() References: <20100304113553.514db067@o> Message-ID: <20100304185405.4436a34c@msiwind> Le Thu, 4 Mar 2010 11:35:53 +0100, spir a ?crit : > > (1) I do not understand an iterable type's __iter__() method to be > compulsary. Actually, each time I have defined one, I had to write: > def __iter__(self): return self > So, I guess that if python does not find __iter__(), but the object > defines next(), then by default the said object could be used as its > own iterator. Explicit is better than implicit, though. In many non-trivial cases, the iterator will have to be a separate object anyway. Also, please note you can implement __iter__ as a generator, which makes things very easy for the simple cases: >>> class C(object): ... def __iter__(self): ... yield 1 ... yield 2 ... >>> c = C() >>> it = iter(c) >>> it >>> next(it) 1 >>> next(it) 2 >>> next(it) Traceback (most recent call last): File "", line 1, in StopIteration >>> list(c) [1, 2] > (3) What I miss actually for iterables (which are their own iterator) > is a kind of __reset__(). No, really, you don't want this, unless you like PHP. As others said, calling iter() again is the well-defined generic way to "reset" your iterable. Then, particular cases can warrant specific APIs, such as file.seek(). Regards Antoine. From grosser.meister.morti at gmx.net Fri Mar 5 05:01:27 2010 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Fri, 05 Mar 2010 05:01:27 +0100 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() In-Reply-To: References: <20100304113553.514db067@o> Message-ID: <4B908217.9060603@gmx.net> On 03/04/2010 10:43 PM, Georg Brandl wrote: >> > (2) But: for any reason next() is not spelled as a "magic" method. If this >> > method becomes the distinctive method of iterables, then it should be called >> > __next__() for consistency. Side-question: Why is it called next(), as it is >> > a magic method for iterators already? > Because it is supposed to be called directly. __iter__() isn't. (This changes > with Python 3, where you have next() as a builtin.) And why is it made a builtin function? What was wrong with it being a normal method? -panzi From andrew at bemusement.org Fri Mar 5 05:17:59 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Fri, 5 Mar 2010 15:17:59 +1100 Subject: [Python-ideas] iterable: next() and __iter__() -- and __reset() In-Reply-To: <4B908217.9060603@gmx.net> References: <20100304113553.514db067@o> <4B908217.9060603@gmx.net> Message-ID: <20100305041759.GH17708@steerpike.home.puzzling.org> Mathias Panzenb?ck wrote: > On 03/04/2010 10:43 PM, Georg Brandl wrote: > >>> (2) But: for any reason next() is not spelled as a "magic" method. If this > >>> method becomes the distinctive method of iterables, then it should be called > >>> __next__() for consistency. Side-question: Why is it called next(), as it is > >>> a magic method for iterators already? > >Because it is supposed to be called directly. __iter__() isn't. (This changes > >with Python 3, where you have next() as a builtin.) > > And why is it made a builtin function? What was wrong with it being a normal method? . -Andrew. From fuzzyman at gmail.com Tue Mar 16 20:11:43 2010 From: fuzzyman at gmail.com (Michael Foord) Date: Tue, 16 Mar 2010 19:11:43 +0000 Subject: [Python-ideas] EuroPython 2010 - Open for registration and reminder of participation Message-ID: <6f4025011003161211r4c78da92q6d02f4ee676e7879@mail.gmail.com> EuroPython 2010 - 17th to 24th July 2010 ---------------------------------------- EuroPython is a conference for the Python programming language community, including the Django, Zope and Plone communities. It is aimed at everyone in the Python community, of all skill levels, both users and programmers. Last year's conference was the largest open source conference in the UK and one of the largest community organised software conferences in Europe. This year EuroPython will be held from the 17th to 24th July in Birmingham, UK. It will include over 100 talks, tutorials, sprints and social events. Registration ------------ Registration is open now at: http://www.europython.eu/registration/ For the best registration rates, book as soon as you can! Extra Early Bird closes soon, after which normal Early Bird rate will apply until 10th May Talks, Activities and Events ---------------------------- Do you have something you wish to present at EuroPython? You want to give a talk, run a tutorial or sprint? Go to http://www.europython.eu/talks/cfp/ for information and advice! Go to http://wiki.europython.eu/Sprints to plan a sprint! Help Us Out ----------- EuroPython is run by volunteers, like you! We could use a hand, and any contribution is welcome. Go to http://wiki.europython.eu/Helping to join us! Go to http://www.europython.eu/contact/ to contact us directly! Sponsors -------- Sponsoring EuroPython is a unique opportunity to affiliate with this prestigious conference and to reach a large number of Python users from computing professionals to academics, from entrepreneurs to motivated and well-educated job seekers. http://www.europython.eu/sponsors/ Spread the Word --------------- We are a community-run not-for-profit conference. Please help to spread the word by distributing this announcement to colleagues, project mailing lists, friends, your blog, Web site, and through your social networking connections. Take a look at our publicity resources: http://wiki.europython.eu/Publicity General Information ------------------- For more information about the conference, please visit the official site: http://www.europython.eu/ Looking forward to see you! The EuroPython Team -- http://www.ironpythoninaction.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Wed Mar 17 08:11:29 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 17 Mar 2010 08:11:29 +0100 Subject: [Python-ideas] Matching multiple regex patterns simultaneously In-Reply-To: <4B903D4B.1050209@canterbury.ac.nz> References: <7659cab31003021239x76ba71cdo4c691f256776f60@mail.gmail.com> <4B903D4B.1050209@canterbury.ac.nz> Message-ID: Greg Ewing, 05.03.2010 00:07: > I have a half-finished project on the back burner > to reimplement the core of Plex using Pyrex, but > it's been languishing for so long that the burner > has probably gone out by now. Maybe I should > relight it... Cython bootstraps the tiny truely performance-critical part of it (Scanners.py) into C code during installation and uses an external Scanners.pxd file to override the Python types in the module with C types. The other modules don't seem to matter in benchmarks, but compiling the inner loop of the scanner clearly gives a performance boost. Stefan From zac256 at gmail.com Wed Mar 17 21:31:20 2010 From: zac256 at gmail.com (Zac Burns) Date: Wed, 17 Mar 2010 13:31:20 -0700 Subject: [Python-ideas] __metafunc__ Message-ID: <333edbe81003171331m6ecfbffet90fb4d5b060e4ad6@mail.gmail.com> I would like to propose a __metafunc__ that would work similar to metaclasses but for functions. My use case is for duck punching all functions defined in a module. This tends to be difficult as a postprocess after __import__ (you have to modify method.im_func and deal with closures and functions in the module namespace that might be references to functions defined elsewhere - it's a mess). If instead when defining a function it did a lookup of __metafunc__ and called it (like a decorator) after a function was defined then I could import the module in a globals that includes my __metafunc__. Feel free to tear this idea apart, improve it, or dash it on the rocks. Perhaps some of you will also see use cases for this or something like it? -- Zachary Burns (407)590-4814 Aim - Zac256FL Production Engineer (Digital Overlord) Zindagi Games -------------- next part -------------- An HTML attachment was scrubbed... URL: From nbvfour at gmail.com Thu Mar 18 07:49:39 2010 From: nbvfour at gmail.com (nbv4) Date: Thu, 18 Mar 2010 02:49:39 -0400 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: References: Message-ID: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> Decorators are great, they make this: def some_func(*args, **kwargs): return "sup" some_func = decorator(some_func) into this: @decorator def some_func(*args, **kwargs): return "sup" which makes the code look more structured. The line at the bottom looks more like it 'belongs' to the function when it's at the top of the def block, as opposed to the bottom. But when it comes to function properties, it'd be nice if that same pattern was availiable: def some_func(*args, **kwargs): return "sup" some_func.printable = True becomes: @printable=True def some_func(*args, **kwargs): return "sup" Any thoughts? From pyideas at rebertia.com Thu Mar 18 08:19:40 2010 From: pyideas at rebertia.com (Chris Rebert) Date: Thu, 18 Mar 2010 00:19:40 -0700 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> Message-ID: <50697b2c1003180019u768b50anb650293376d6a88f@mail.gmail.com> On Wed, Mar 17, 2010 at 11:49 PM, nbv4 wrote: > Decorators are great, they make this: > > def some_func(*args, **kwargs): > ? ?return "sup" > some_func = decorator(some_func) > > into this: > > @decorator > def some_func(*args, **kwargs): > ? ?return "sup" > > which makes the code look more structured. The line at the bottom > looks more like it 'belongs' to the function when it's at the top of > the def block, as opposed to the bottom. > > But when it comes to function properties, it'd be nice if that same > pattern was availiable: > > def some_func(*args, **kwargs): > ? ?return "sup" > some_func.printable = True > > becomes: > > @printable=True > def some_func(*args, **kwargs): > ? ?return "sup" > > Any thoughts? Are function attributes used *that* much? Cheers, Chris -- http://blog.rebertia.com From anfedorov at gmail.com Thu Mar 18 08:34:02 2010 From: anfedorov at gmail.com (Andrey Fedorov) Date: Thu, 18 Mar 2010 03:34:02 -0400 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> Message-ID: <7659cab31003180034t5ef10b1fp1c663d74c9aae566@mail.gmail.com> On Thu, Mar 18, 2010 at 2:49 AM, nbv4 wrote: > @printable=True > def some_func(*args, **kwargs): > return "sup" > > Any thoughts? > If you have that many properties, there's no need to extend the syntax, just do something like: def prop(**props): def wee(f): f.__dict__.update(props) return f return wee @prop(printable=True, other_prop="yo!") def some_func(): return "sup, %s" % some_func.other_prop print some_func() # prints "sup, yo!" -------------- next part -------------- An HTML attachment was scrubbed... URL: From jafo at tummy.com Thu Mar 18 13:08:47 2010 From: jafo at tummy.com (Sean Reifschneider) Date: Thu, 18 Mar 2010 06:08:47 -0600 Subject: [Python-ideas] Adding exception logging function to syslog module. Message-ID: <20100318120847.GA8871@tummy.com> As someone who writes mostly programs that run unattended on servers, one thing I often want to do is have my tracebacks logged to syslog. I would propose adding something like the following to the syslog module: def logexceptions(also_stderr = True): class ExceptHook: def __init__(self, useStderr = False): self.useStderr = useStderr def __call__(self, etype, evalue, etb): import traceback, string tb = traceback.format_exception(*(etype, evalue, etb)) tb = map(string.rstrip, tb) tb = string.join(tb, '\n') for line in string.split(tb, '\n'): syslog.syslog(line) if self.useStderr: sys.stderr.write(line + '\n') sys.excepthook = ExceptHook(also_stderr) Now, the downside to this is that currently the syslog module is entirely in C. So either I'd need to implement the above in C, or I'd need to add a Python wrapper to the C module and use that. This would allow users to also log exceptions to syslog by doing: import syslog syslog.openlog('myprogram', syslog.LOG_PID, syslog.LOG_MAIL) syslog.logexceptions() Thoughts? Thanks, Sean -- Python: even though everyone's using it now, somehow it's still the coolest. Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability From denis.spir at gmail.com Thu Mar 18 19:32:05 2010 From: denis.spir at gmail.com (spir) Date: Thu, 18 Mar 2010 19:32:05 +0100 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <50697b2c1003180019u768b50anb650293376d6a88f@mail.gmail.com> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> <50697b2c1003180019u768b50anb650293376d6a88f@mail.gmail.com> Message-ID: <20100318193205.0ffc7978@o> On Thu, 18 Mar 2010 00:19:40 -0700 Chris Rebert wrote: > Are function attributes used *that* much? I use them as a nice (& explicit, & clear) alternative to closures, esp. for func factories, eg: def powerN(n): def f(x): return x ** f.n f.n = n return f power3 = powerN(3) for i in range(1,10): print "%s:%s" %(i,power3(i)), # ==> 1:1 2:8 3:27 4:64 5:125 6:216 7:343 8:512 9:729 I find this conceptually much more satisfying than the following, since the parameter really is an attribute of the function (also, like a real upvalue, it can be explicetely updated). def powerN(n): def f(x,n=n): # ugly ;-) return x ** n return f (Let us use the opportunity that python funcs are real objects :-) A "parameterisable" generator factory: def powers(n, start=1, stop=None): def f(): while True: f.i += 1 if f.stop and f.i >= f.stop: raise StopIteration yield (f.i, f.i ** f.n) f.n = n f.i = start-1 f.stop = stop return f for (i,x) in powers(3, 1,10)(): print "%s:%s" %(i,x), # ==> 1:1 2:8 3:27 4:64 5:125 6:216 7:343 8:512 9:729 Denis ________________________________ vit e estrany spir.wikidot.com From python at mrabarnett.plus.com Thu Mar 18 19:56:43 2010 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 18 Mar 2010 18:56:43 +0000 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <20100318193205.0ffc7978@o> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> <50697b2c1003180019u768b50anb650293376d6a88f@mail.gmail.com> <20100318193205.0ffc7978@o> Message-ID: <4BA2776B.8080206@mrabarnett.plus.com> spir wrote: > On Thu, 18 Mar 2010 00:19:40 -0700 > Chris Rebert wrote: > >> Are function attributes used *that* much? > > I use them as a nice (& explicit, & clear) alternative to closures, esp. for func factories, eg: > > def powerN(n): > def f(x): > return x ** f.n > f.n = n > return f > power3 = powerN(3) > for i in range(1,10): > print "%s:%s" %(i,power3(i)), > # ==> 1:1 2:8 3:27 4:64 5:125 6:216 7:343 8:512 9:729 > > I find this conceptually much more satisfying than the following, since the parameter really is an attribute of the function (also, like a real upvalue, it can be explicetely updated). > > def powerN(n): > def f(x,n=n): # ugly ;-) > return x ** n > return f > [snip] This also works: def powerN(n): def f(x): return x ** n return f From anfedorov at gmail.com Thu Mar 18 22:01:55 2010 From: anfedorov at gmail.com (Andrey Fedorov) Date: Thu, 18 Mar 2010 17:01:55 -0400 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <20100318193205.0ffc7978@o> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> <50697b2c1003180019u768b50anb650293376d6a88f@mail.gmail.com> <20100318193205.0ffc7978@o> Message-ID: <7659cab31003181401r11580368t8a40468d4399254c@mail.gmail.com> spir wrote: > A "parameterisable" generator factory: [...] > Good lord, no! Nooo! "power" is one thing with one semantic meaning "range" is another thing with another semantic meaning There is absolutely no excuse for mixing the two implementation into a "power-range" monstrosity. Implement the two separately and make some custom function composition like: def pow(x, n): return x**n def pow_o_range(n, a, b): return [(x, pow(x, n)) for x in range(a,b)] for i, x in pow_o_range(3, 1, 10): print "%s:%s" % (i,x), # prints "1:1 2:8 3:27 4:64 5:125 6:216 7:343 8:512 9:729" -------------- next part -------------- An HTML attachment was scrubbed... URL: From nbvfour at gmail.com Thu Mar 18 23:02:52 2010 From: nbvfour at gmail.com (nbv4) Date: Thu, 18 Mar 2010 15:02:52 -0700 (PDT) Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <7659cab31003181401r11580368t8a40468d4399254c@mail.gmail.com> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> <50697b2c1003180019u768b50anb650293376d6a88f@mail.gmail.com> <20100318193205.0ffc7978@o> <7659cab31003181401r11580368t8a40468d4399254c@mail.gmail.com> Message-ID: <6ddd86bc-2aa4-4a98-bf4f-d87bfb771db1@q16g2000yqq.googlegroups.com> On Mar 18, 5:01?pm, Andrey Fedorov wrote: > spir wrote: > > A "parameterisable" generator factory: [...] > > Good lord, no! Nooo! > > "power" is one thing with one semantic meaning > "range" is another thing with another semantic meaning > > There is absolutely no excuse for mixing the two implementation into a > "power-range" monstrosity. Implement the two separately and make some custom > function composition like: > > def pow(x, n): > ? ? return x**n > > def pow_o_range(n, a, b): > ? ? return [(x, pow(x, n)) for x in range(a,b)] > > for i, x in pow_o_range(3, 1, 10): > ? ?print "%s:%s" % (i,x), > > # prints "1:1 2:8 3:27 4:64 5:125 6:216 7:343 8:512 9:729" > i was just an example to demonstrate the usage dude, :) From greg.ewing at canterbury.ac.nz Fri Mar 19 02:50:56 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 19 Mar 2010 14:50:56 +1300 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <20100318193205.0ffc7978@o> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> <50697b2c1003180019u768b50anb650293376d6a88f@mail.gmail.com> <20100318193205.0ffc7978@o> Message-ID: <4BA2D880.9010908@canterbury.ac.nz> spir wrote: > def powerN(n): > def f(x,n=n): # ugly ;-) > return x ** n > return f Since nothing can change the value of n there after powerN returns, there's no need for default argument abuse. You can just write def powerN(n): def f(x): return x ** n return f -- Greg From dstanek at dstanek.com Fri Mar 19 03:28:46 2010 From: dstanek at dstanek.com (David Stanek) Date: Thu, 18 Mar 2010 22:28:46 -0400 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> Message-ID: On Thu, Mar 18, 2010 at 2:49 AM, nbv4 wrote: > > @printable=True > def some_func(*args, **kwargs): > ? ?return "sup" > > Any thoughts? > I don't remember seeing much code using function properties. I would probably just create a new decorator for this. >>> def add_props(**kwargs): ... def _(func): ... for key in kwargs: ... setattr(func, key, kwargs[key]) ... return func ... return _ ... >>> @add_props(printable=True) ... def f(): ... """Example function""" ... >>> f.printable True >>> -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek From fdrake at acm.org Fri Mar 19 03:33:40 2010 From: fdrake at acm.org (Fred Drake) Date: Thu, 18 Mar 2010 22:33:40 -0400 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> Message-ID: <9cee7ab81003181933g37d88a82g973c4d8b3d4369e7@mail.gmail.com> On Thu, Mar 18, 2010 at 10:28 PM, David Stanek wrote: > I don't remember seeing much code using function properties. I would > probably just create a new decorator for this. I've not either, but if we determine that this is a useful facility to include in the standard library, I'd be in favor of something like this: @setattr(printable=True) def f(): """Example function""" -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From scott+python-ideas at scottdial.com Fri Mar 19 04:43:34 2010 From: scott+python-ideas at scottdial.com (Scott Dial) Date: Thu, 18 Mar 2010 23:43:34 -0400 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <9cee7ab81003181933g37d88a82g973c4d8b3d4369e7@mail.gmail.com> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> <9cee7ab81003181933g37d88a82g973c4d8b3d4369e7@mail.gmail.com> Message-ID: <4BA2F2E6.9020204@scottdial.com> On 3/18/2010 10:33 PM, Fred Drake wrote: > On Thu, Mar 18, 2010 at 10:28 PM, David Stanek wrote: >> I don't remember seeing much code using function properties. I would >> probably just create a new decorator for this. > > I've not either, but if we determine that this is a useful facility to > include in the standard library, I'd be in favor of something like > this: > > @setattr(printable=True) > def f(): > """Example function""" > For the sake of a head count, I have used function attributes for the purpose of memoizing computations (both return values and intermediate calculations). A representative example: @setattr(primes=set(), composites=set()) def is_prime(n): if n <= 0: raise ValueError(n) elif n == 1: return True elif n in is_prime.primes: return True elif n in is_prime.composites: return False for p in is_prime.primes: if n % p == 0: is_prime.composites.add(n) return False is_prime.primes.add(n) return True But honestly, the biggest problem I have with this pattern is that you explicitly have to say the name of the function since there is no "self" equivalent. I think it would be more maintenance-friendly to do: def _(n): primes = set() composites = set() def is_prime(n): if n <= 0: raise ValueError(n) elif n == 1: return True elif n in primes: return True elif n in composites: return False for p in primes: if n % p == 0: composites.add(n) return False primes.add(n) return True return is_prime is_prime = _() Which is to say, I would be more amenable to adding such a feature if it was building closure. And in that usage, it would seem to be more generally useful and well-behaved.. outsiders don't normally modify closures (if you want this sort of interface, you probably should define a class).. it allows you to avoid naming the function in the function.. and it is a much closer substitute for the default arguments kludge that is so commonly abused. @closure(primes=set(), composites=set()) def is_prime(n): if n <= 0: raise ValueError(n) elif n == 1: return True elif n in primes: return True elif n in composites: return False for p in primes: if n % p == 0: composites.add(n) return False primes.add(n) return True But, that's just my two cents. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From arnodel at googlemail.com Fri Mar 19 09:25:09 2010 From: arnodel at googlemail.com (Arnaud Delobelle) Date: Fri, 19 Mar 2010 08:25:09 +0000 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <4BA2F2E6.9020204@scottdial.com> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> <9cee7ab81003181933g37d88a82g973c4d8b3d4369e7@mail.gmail.com> <4BA2F2E6.9020204@scottdial.com> Message-ID: On 19 Mar 2010, at 03:43, Scott Dial wrote: > On 3/18/2010 10:33 PM, Fred Drake wrote: >> On Thu, Mar 18, 2010 at 10:28 PM, David Stanek >> wrote: >>> I don't remember seeing much code using function properties. I would >>> probably just create a new decorator for this. >> >> I've not either, but if we determine that this is a useful facility >> to >> include in the standard library, I'd be in favor of something like >> this: >> >> @setattr(printable=True) >> def f(): >> """Example function""" >> > > For the sake of a head count, I have used function attributes for the > purpose of memoizing computations (both return values and intermediate > calculations). A representative example: > > @setattr(primes=set(), composites=set()) > def is_prime(n): > if n <= 0: > raise ValueError(n) > elif n == 1: > return True > elif n in is_prime.primes: > return True > elif n in is_prime.composites: > return False > for p in is_prime.primes: > if n % p == 0: > is_prime.composites.add(n) > return False > is_prime.primes.add(n) > return True > > But honestly, the biggest problem I have with this pattern is that you > explicitly have to say the name of the function since there is no > "self" > equivalent. I think it would be more maintenance-friendly to do: > > def _(n): > primes = set() > composites = set() > def is_prime(n): > if n <= 0: > raise ValueError(n) > elif n == 1: > return True > elif n in primes: > return True > elif n in composites: > return False > for p in primes: > if n % p == 0: > composites.add(n) > return False > primes.add(n) > return True > return is_prime > is_prime = _() > > Which is to say, I would be more amenable to adding such a feature > if it > was building closure. And in that usage, it would seem to be more > generally useful and well-behaved.. outsiders don't normally modify > closures (if you want this sort of interface, you probably should > define > a class).. it allows you to avoid naming the function in the > function.. > and it is a much closer substitute for the default arguments kludge > that > is so commonly abused. > > @closure(primes=set(), composites=set()) > def is_prime(n): > if n <= 0: > raise ValueError(n) > elif n == 1: > return True > elif n in primes: > return True > elif n in composites: > return False > for p in primes: > if n % p == 0: > composites.add(n) > return False > primes.add(n) > return True > > But, that's just my two cents. > > -- > Scott Dial > scott at scottdial.com > scodial at cs.indiana.edu > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas Why not just use default arguments? Arnaud From scott+python-ideas at scottdial.com Fri Mar 19 10:07:29 2010 From: scott+python-ideas at scottdial.com (Scott Dial) Date: Fri, 19 Mar 2010 05:07:29 -0400 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> <9cee7ab81003181933g37d88a82g973c4d8b3d4369e7@mail.gmail.com> <4BA2F2E6.9020204@scottdial.com> Message-ID: <4BA33ED1.2090409@scottdial.com> On 3/19/2010 4:25 AM, Arnaud Delobelle wrote: > On 19 Mar 2010, at 03:43, Scott Dial > wrote: >> Which is to say, I would be more amenable to adding such a feature if it >> was building closure. And in that usage, it would seem to be more >> generally useful and well-behaved.. outsiders don't normally modify >> closures (if you want this sort of interface, you probably should define >> a class).. it allows you to avoid naming the function in the function.. >> and it is a much closer substitute for the default arguments kludge that >> is so commonly abused. >> > > Why not just use default arguments? > My problem with this particular kludge is that rarely are these arguments meaningful as *arguments*. They become a part of the function definition that shows up in documentation. Worse to me is that the keyword names block those names from appearing in **kwds, which is problematic for creating a transparent function (e.g., when writing a decorator that takes an unknown set of keyword arguments) and evokes dynamic-scoping madness. Additionally, they are a source of subtle errors when extra arguments are passed to functions (although this is mitigated mostly by Py3's keyword-only arguments syntax). All of those problems go away with a magical @closure() decorator, but I realize that creating such a decorator would be quite a kludge in-and-of-itself. Perhaps it's a bit too clever. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From arnodel at googlemail.com Fri Mar 19 14:15:20 2010 From: arnodel at googlemail.com (Arnaud Delobelle) Date: Fri, 19 Mar 2010 13:15:20 +0000 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <4BA33ED1.2090409@scottdial.com> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> <9cee7ab81003181933g37d88a82g973c4d8b3d4369e7@mail.gmail.com> <4BA2F2E6.9020204@scottdial.com> <4BA33ED1.2090409@scottdial.com> Message-ID: <9bfc700a1003190615u721d0e94x12875914c61cf9a6@mail.gmail.com> On 19 March 2010 09:07, Scott Dial wrote: > On 3/19/2010 4:25 AM, Arnaud Delobelle wrote: >> On 19 Mar 2010, at 03:43, Scott Dial >> wrote: >>> Which is to say, I would be more amenable to adding such a feature if it >>> was building closure. And in that usage, it would seem to be more >>> generally useful and well-behaved.. outsiders don't normally modify >>> closures (if you want this sort of interface, you probably should define >>> a class).. it allows you to avoid naming the function in the function.. >>> and it is a much closer substitute for the default arguments kludge that >>> is so commonly abused. >>> >> >> Why not just use default arguments? >> > > My problem with this particular kludge is that rarely are these > arguments meaningful as *arguments*. They become a part of the function > definition that shows up in documentation. Worse to me is that the > keyword names block those names from appearing in **kwds, which is > problematic for creating a transparent function (e.g., when writing a > decorator that takes an unknown set of keyword arguments) and evokes > dynamic-scoping madness. Additionally, they are a source of subtle > errors when extra arguments are passed to functions (although this is > mitigated mostly by Py3's keyword-only arguments syntax). > > All of those problems go away with a magical @closure() decorator, but I > realize that creating such a decorator would be quite a kludge > in-and-of-itself. Perhaps it's a bit too clever. I wrote such a decorator a while ago, I called it 'bind'. It was indeed quite a kludge, but if you're interested I can post it. It worked by rewriting the bytecode of the function it decorated. If I remember correctly, it has some limitations, e.g. it didn't work with local functions i.e. @bind(x=3) def f(): def g(): return x return g f()() would return the current value of the global variable 'x' rather than 3. I don't think I got round to solving this (it would require looking at the constants in the code which are themselves code objects an apply the same bytecode transformation recursively). Also, it was for Python 2.X so I don't know if it would work as-is in Python 3. -- Arnaud From jimjjewett at gmail.com Fri Mar 19 15:57:42 2010 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 19 Mar 2010 10:57:42 -0400 Subject: [Python-ideas] setting function properties with a decorator syntax In-Reply-To: <20100318193205.0ffc7978@o> References: <626451591003172349q307789f3v43c62def59692c7c@mail.gmail.com> <50697b2c1003180019u768b50anb650293376d6a88f@mail.gmail.com> <20100318193205.0ffc7978@o> Message-ID: For anyone finding this on a later search, note that he still uses a closure; just a slightly neater one. Within a function, there is no way to refer to the function itself, except by name -- and that name may well have been reassigned to something else. A more local enclosure protects against that reassignment. -jJ On 3/18/10, spir wrote: > On Thu, 18 Mar 2010 00:19:40 -0700 > Chris Rebert wrote: > >> Are function attributes used *that* much? > > I use them as a nice (& explicit, & clear) alternative to closures, esp. for > func factories, eg: > > def powerN(n): > def f(x): > return x ** f.n > f.n = n > return f > power3 = powerN(3) > for i in range(1,10): > print "%s:%s" %(i,power3(i)), > # ==> 1:1 2:8 3:27 4:64 5:125 6:216 7:343 8:512 9:729 > > I find this conceptually much more satisfying than the following, since the > parameter really is an attribute of the function (also, like a real upvalue, > it can be explicetely updated). > > def powerN(n): > def f(x,n=n): # ugly ;-) > return x ** n > return f > > (Let us use the opportunity that python funcs are real objects :-) > > A "parameterisable" generator factory: > > def powers(n, start=1, stop=None): > def f(): > while True: > f.i += 1 > if f.stop and f.i >= f.stop: > raise StopIteration > yield (f.i, f.i ** f.n) > f.n = n > f.i = start-1 > f.stop = stop > return f > for (i,x) in powers(3, 1,10)(): > print "%s:%s" %(i,x), > # ==> 1:1 2:8 3:27 4:64 5:125 6:216 7:343 8:512 9:729 > > > Denis > ________________________________ > > vit e estrany > > spir.wikidot.com > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From eric at trueblade.com Tue Mar 23 00:31:08 2010 From: eric at trueblade.com (Eric Smith) Date: Mon, 22 Mar 2010 19:31:08 -0400 Subject: [Python-ideas] Adding exception logging function to syslog module. In-Reply-To: <20100318120847.GA8871@tummy.com> References: <20100318120847.GA8871@tummy.com> Message-ID: <4BA7FDBC.3060808@trueblade.com> Sean Reifschneider wrote: > As someone who writes mostly programs that run unattended on servers, one > thing I often want to do is have my tracebacks logged to syslog. I would > propose adding something like the following to the syslog module: > > def logexceptions(also_stderr = True): > class ExceptHook: > def __init__(self, useStderr = False): > self.useStderr = useStderr > > def __call__(self, etype, evalue, etb): > import traceback, string > tb = traceback.format_exception(*(etype, evalue, etb)) > tb = map(string.rstrip, tb) > tb = string.join(tb, '\n') > for line in string.split(tb, '\n'): > syslog.syslog(line) > if self.useStderr: > sys.stderr.write(line + '\n') > sys.excepthook = ExceptHook(also_stderr) > > Now, the downside to this is that currently the syslog module is entirely > in C. So either I'd need to implement the above in C, or I'd need to add a > Python wrapper to the C module and use that. > > This would allow users to also log exceptions to syslog by doing: > > import syslog > syslog.openlog('myprogram', syslog.LOG_PID, syslog.LOG_MAIL) > syslog.logexceptions() > > Thoughts? I think this would be a great addition. I have to do similar things all of the time, although I never though of using sys.excepthook, for some reason. I'll have to steal your code in the meantime! A simpler pure Python version would be: from __future__ import print_function def logexceptions(also_stderr=True): import sys import traceback import syslog def syslog_exception(etype, evalue, etb): # The result of traceback.format_exception might contain # embedded newlines, so we have the nested loops. for line in traceback.format_exception(etype, evalue, etb): for line in line.rstrip().split('\n'): syslog.syslog(line) if also_stderr: print(line, file=sys.stderr) sys.excepthook = syslog_exception logexceptions(True) If you need any help implementing this in C, I'll assist (although I'm out of town for 2 weeks starting Thursday). -- Eric. From eric at trueblade.com Tue Mar 23 21:24:41 2010 From: eric at trueblade.com (Eric Smith) Date: Tue, 23 Mar 2010 16:24:41 -0400 Subject: [Python-ideas] Adding exception logging function to syslog module. In-Reply-To: <4BA7FDBC.3060808@trueblade.com> References: <20100318120847.GA8871@tummy.com> <4BA7FDBC.3060808@trueblade.com> Message-ID: <4BA92389.7060401@trueblade.com> Eric Smith wrote: > I think this would be a great addition. I have to do similar things all > of the time, although I never though of using sys.excepthook, for some > reason. I'll have to steal your code in the meantime! > > A simpler pure Python version would be: > > from __future__ import print_function > > def logexceptions(also_stderr=True): > import sys > import traceback > import syslog > def syslog_exception(etype, evalue, etb): > # The result of traceback.format_exception might contain > > # embedded newlines, so we have the nested loops. > > for line in traceback.format_exception(etype, evalue, etb): > for line in line.rstrip().split('\n'): > syslog.syslog(line) > if also_stderr: > print(line, file=sys.stderr) > sys.excepthook = syslog_exception > > logexceptions(True) On second thought, maybe it would be better to optionally chain to the existing sys.excepthook instead of assuming it writes to stderr. The above code would become: def logexceptions(chain=True): import sys import traceback import syslog current_hook = sys.excepthook def syslog_exception(etype, evalue, etb): if chain: current_hook(etype, evalue, etb) # The result of traceback.format_exception might contain # embedded newlines, so we have the nested loops. for line in traceback.format_exception(etype, evalue, etb): for line in line.rstrip().split('\n'): syslog.syslog(line) sys.excepthook = syslog_exception I think I'll open an issue for this. I don't think the language moratorium would preclude its addition. Eric. -- Eric. From jafo at tummy.com Wed Mar 24 22:57:42 2010 From: jafo at tummy.com (Sean Reifschneider) Date: Wed, 24 Mar 2010 15:57:42 -0600 Subject: [Python-ideas] Adding exception logging function to syslog module. In-Reply-To: <4BA92389.7060401@trueblade.com> References: <20100318120847.GA8871@tummy.com> <4BA7FDBC.3060808@trueblade.com> <4BA92389.7060401@trueblade.com> Message-ID: <4BAA8AD6.3020807@tummy.com> On 03/23/2010 02:24 PM, Eric Smith wrote: > On second thought, maybe it would be better to optionally chain to the > existing sys.excepthook instead of assuming it writes to stderr. The > above code would become: That looks just great. I understand you're leaving tomorrow, so I'll probably try converting it to C while you're away. I expect I should be able to find some time. I'll make an issue for it if you haven't already, and submit it for review there. I don't imagine it will be too hard, but I haven't really done any C API hacking at the level of creating an inner function. I should be able to figure it out though. Thanks, and have a good trip. Sean -- Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature URL: From eric at trueblade.com Wed Mar 24 23:24:46 2010 From: eric at trueblade.com (Eric Smith) Date: Wed, 24 Mar 2010 18:24:46 -0400 Subject: [Python-ideas] Adding exception logging function to syslog module. In-Reply-To: <4BAA8AD6.3020807@tummy.com> References: <20100318120847.GA8871@tummy.com> <4BA7FDBC.3060808@trueblade.com> <4BA92389.7060401@trueblade.com> <4BAA8AD6.3020807@tummy.com> Message-ID: <4BAA912E.7010501@trueblade.com> Sean Reifschneider wrote: > On 03/23/2010 02:24 PM, Eric Smith wrote: >> On second thought, maybe it would be better to optionally chain to the >> existing sys.excepthook instead of assuming it writes to stderr. The >> above code would become: > > That looks just great. I understand you're leaving tomorrow, so I'll > probably try converting it to C while you're away. I expect I should be > able to find some time. I'll make an issue for it if you haven't already, > and submit it for review there. > > I don't imagine it will be too hard, but I haven't really done any C API > hacking at the level of creating an inner function. I should be able to > figure it out though. > > Thanks, and have a good trip. Thanks, Sean. It's issue 8214, you should already be nosy on it. I used a slightly different method there, designed to remove a reference to sys.excepthook if not chaining, but the principle is the same. I'm not sure about the closure aspect of this in C. You could do it with a global variable, but I'm not wild about that. Maybe you can ask on python-dev for suggestions. I'll be checking email occasionally while I'm away. -- Eric. From jafo at tummy.com Wed Mar 24 23:34:52 2010 From: jafo at tummy.com (Sean Reifschneider) Date: Wed, 24 Mar 2010 16:34:52 -0600 Subject: [Python-ideas] Adding exception logging function to syslog module. In-Reply-To: <4BAA912E.7010501@trueblade.com> References: <20100318120847.GA8871@tummy.com> <4BA7FDBC.3060808@trueblade.com> <4BA92389.7060401@trueblade.com> <4BAA8AD6.3020807@tummy.com> <4BAA912E.7010501@trueblade.com> Message-ID: <4BAA938C.7000003@tummy.com> On 03/24/2010 04:24 PM, Eric Smith wrote: > I'm not sure about the closure aspect of this in C. You could do it with > a global variable, but I'm not wild about that. Maybe you can ask on > python-dev for suggestions. I'll probably do some digging into the API to see about creating a function via C, and from that the issue of the closure will probably become more clear. If not, python-dev will be my go-to-guy. :-) Sean -- Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature URL: From alexander.belopolsky at gmail.com Thu Mar 25 07:50:41 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 25 Mar 2010 02:50:41 -0400 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> Message-ID: Since we are now venturing in the realm of python-ideas, I am switching the mailing lists. Please see more below. On Wed, Mar 24, 2010 at 7:31 PM, Mark Dickinson wrote: > On Wed, Mar 24, 2010 at 11:11 PM, Alexander Belopolsky > wrote: >> On Wed, Mar 24, 2010 at 7:02 PM, Mark Dickinson wrote: >> .. >>> >>> So if I understand correctly, you propose that float('nan') == >>> float('nan') return True. ?Would you also suggest extending this >>> behaviour to Decimal nans? >>> >> >> yes >> > > Okay. ?So now it seems to me that there are many decisions to make: I maybe show my ignorance, but I don't see the significance of bringing Decimal nans to the discussion. My answers below apply equally to binary and decimal floating point numbers. > should any Decimal nan compare equal to any other? Yes. >?What if the two nans have different payloads or signs? They are still equal. Just as 0.0 and -0.0 are now. >?How about comparing a > signaling nan with either an arbitrary quiet nan, or with the exact > quiet nan that corresponds to the signaling nan? I don't think signaling nans can be produced using Python float type. I may be mistaken about this and the answer may be different in some decimal contexts. Nevertheless, I always thought that any operation with an sNaN should raise an exception, so comparing an sNaN with anything, NaN or not, should just raise an exception. > ?How do decimal nans compare with float nans? Good question. Does IEEE 754 or rather 854 specify cross-radix comparisons? My answer would be that they should compare equal. > ?Should Decimal.compare(Decimal('nan'), Decimal('nan')) return 0 rather than nan? I would think with deprecation of __cmp__, Decimal.compare would not be that useful. I would rather not expand this discussion to what NaN < NaN should return. In my view it should just raise an exception because there is no practical way to propagate NaN through boolean results. Decimal.compare may similarly raise an exception when comparing NaNs. Returning Decimal('nan') seems least useful. With the current NaN properties, I would rather see Decimal.compare return 2 for unordered values. >?If not, how do you justify the difference between == and compare? Just as (a > b or b < a) is currently not equivalent to a != b in the presence of unordered values (i.e. when a and b are of different type). > ?If so, how do you justify the > deviation from the standard on which the decimal modulo is based? I assume you mean "decimal module" because I don't know what "decimal modulo" is based on. Is that IEEE 854? I need to get a copy of that before I can answer. :-) > > In answering all these questions, you effectively end up developing > your own standard, and hoping that all the answers you chose are > sensible, consistent, and useful. > As Grace Hopper is reported to have said, "The wonderful thing about standards is that there are so many of them to choose from." As much as I don't like most of the choices that Java designers made, I think they got this one right. > Alternatively, we could do what we're currently doing: ?make use of > *existing* standards to answer these questions, and rely on the > expertise of the many who've thought about this in depth. > I don't think I can answer this better than Bertrand Meyer: """ Before commenting further let me note the obvious: I am by no means a numerics expert; I know that IEEE 754 was a tremendous advance, and that it was designed by some of the best minds in the field, headed by Velvel Kahan who received a Turing Award in part for that success. So it is quite possible that I am about to bury myself in piles of ridicule. On the other hand I have also learned that (1) ridicule does not kill, so I am game; and more importantly (2) experts are often right but not always, and it is always proper to question their reasoning. So without fear let me not stop at the arguments that ?the committee must have voted on this point and they obviously knew what they were doing? and ?it is the standard and implemented on zillions of machines, you cannot change it now?. Both are true enough, but not an excuse to censor questions. """ http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/ > You say that you don't see any compromise: ?I say that there's value > in adhering to (de facto and de jure) standards, and I see a > compromise between standards adherence and Python pragmatics. Bertrand Meyer again: """ A few of us who had to examine the issue recently think that ? whatever the standard says at the machine level ? a programming language should support the venerable properties that equality is reflexive and that assignment yields equality. """ IEEE standards were developed in a different problem domain: hardware or low level programming language design. They may not be appropriate for an object oriented language like Python. Java and recently Eiffel designers seem to have realized that. From denis.spir at gmail.com Thu Mar 25 11:48:00 2010 From: denis.spir at gmail.com (spir =?UTF-8?B?4pij?=) Date: Thu, 25 Mar 2010 11:48:00 +0100 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> Message-ID: <20100325114800.072bfa1a@o> On Thu, 25 Mar 2010 02:50:41 -0400 Alexander Belopolsky wrote: > Bertrand Meyer again: > > """ > A few of us who had to examine the issue recently think that ? > whatever the standard says at the machine level ? a programming > language should support the venerable properties that equality is > reflexive and that assignment yields equality. > """ > > IEEE standards were developed in a different problem domain: hardware > or low level programming language design. They may not be > appropriate for an object oriented language like Python. Java and > recently Eiffel designers seem to have realized that. Hum, should the above be interpreted as: a = NAN ==> not only a is NAN but also a == NAN and further: b = a ==> b == a ? (Else there should be a distinction between equality assignment and identity assignemt? b = a # ==> a is b and a == b b := a # ==> a is b and possibly a == b ;-) Denis ________________________________ vit esse estrany ? spir.wikidot.com From masklinn at masklinn.net Thu Mar 25 12:28:22 2010 From: masklinn at masklinn.net (Masklinn) Date: Thu, 25 Mar 2010 12:28:22 +0100 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> Message-ID: <83DAEA2C-05CF-45EC-BCE7-0B30490DDEC1@masklinn.net> On 25 Mar 2010, at 07:50 , Alexander Belopolsky wrote: > > Since we are now venturing in the realm of python-ideas, I am > switching the mailing lists. Please see more below. > > On Wed, Mar 24, 2010 at 7:31 PM, Mark Dickinson wrote: >> On Wed, Mar 24, 2010 at 11:11 PM, Alexander Belopolsky >> wrote: >>> On Wed, Mar 24, 2010 at 7:02 PM, Mark Dickinson wrote: >>> .. >>>> >>>> So if I understand correctly, you propose that float('nan') == >>>> float('nan') return True. Would you also suggest extending this >>>> behaviour to Decimal nans? >>>> >>> >>> yes >>> >> >> Okay. So now it seems to me that there are many decisions to make: > > I maybe show my ignorance, but I don't see the significance of > bringing Decimal nans to the discussion. My answers below apply > equally to binary and decimal floating point numbers. > >> should any Decimal nan compare equal to any other? > > Yes. > >> What if the two nans have different payloads or signs? > > They are still equal. Just as 0.0 and -0.0 are now. Excuse me but NaNs propagate right? So we'd have not only float('nan') == float('nan'), but also float('nan') = float('nan') + 1 right? Indeed, any operation on a NaN would return a result equal (and identical?) to itself. I'm not sure that makes more sense than the current behavior (which is at least standard and specified). > IEEE standards were developed in a different problem domain: hardware > or low level programming language design. I don't really agree. They were developed in a problem domain of mapping abstract mathematical concepts to computing. And it's not like they're static and never updated (indeed, IEEE-754 was updated in 2008, if anyone has an IEEE login it would be possible to check the purposes and assumptions the standard set for itself). I believe changing the behavior of nan to have "nan == nan" would be broken, nan is not a "normal" value (or set of values) of the space. Having the option of getting an exception upon getting a nan on the other hand (as the default behavior even, with a switch to the current behavior using contexts similar to decimal's), why not. > http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/ It's interesting that Bertrand starts from > I am talking about a value, as in mathematics, not an expression, as in programming but doesn't seem to consider that, in the first place, the very concept of NaN is due to the limitations of physical space compared to the abstract mathematical concepts involved. The very idea of IEEE-754 floats are a a definite departure from the fundamental laws of mathematics, making Bertrand's final request > It is rather dangerous indeed to depart from the fundamental laws of mathematics. very weird. Unless you remove floats from your language, you're not dealing with grounds mathematics cover in the first place, you're dealing with a concretization of mathematical ideas, and as all changes in abstraction levels, it will leak one way or another. I'd support "fixing" nan behaviors, but not by making operations on nan reflexive (and nan a normal value, which it isn't) and instead by treating all of them as errors, the same way calling an arbitrary attribute no None results in an AttributeError in Python. From alexander.belopolsky at gmail.com Thu Mar 25 19:06:47 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 25 Mar 2010 14:06:47 -0400 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> Message-ID: > On Wed, Mar 24, 2010 at 7:31 PM, Mark Dickinson wrote: .. >>?What if the two nans have different payloads or signs? > > They are still equal. ?Just as 0.0 and -0.0 are now. > Interestingly, Java departs from IEEE 754 on that last point as well: """ Note that in most cases, for two instances of class Double, d1 and d2, the value of d1.equals(d2) is true if and only if d1.doubleValue() == d2.doubleValue() also has the value true. However, there are two exceptions: If d1 and d2 both represent Double.NaN, then the equals method returns true, even though Double.NaN==Double.NaN has the value false. If d1 represents +0.0 while d2 represents -0.0, or vice versa, the equal test has the value false, even though +0.0==-0.0 has the value true. This allows hashtables to operate properly. """ http://java.sun.com/j2se/1.3/docs/api/java/lang/Double.html From rhamph at gmail.com Thu Mar 25 19:33:48 2010 From: rhamph at gmail.com (Adam Olsen) Date: Thu, 25 Mar 2010 12:33:48 -0600 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> Message-ID: IMO, we should go for the simplest of all usage: a NaN singleton, shared by float and Decimal, compares equal to itself, unsortable, no payload. Usage of the payload is hypothetical. Situations where comparing false is "more correct" than comparing true are hypothetical, and grossly outweighed by the situations where comparing true is blatantly correct. IEEE 754 doesn't say NaN should compare false; it says it should compare "unordered". We can't do that as we're not using a 4-way comparison (which wouldn't generalize to complex anyway), so we have to make up our own solution. It might as well fit our needs, rather than nobodies. -- Adam Olsen, aka Rhamphoryncus From dickinsm at gmail.com Thu Mar 25 19:58:12 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 25 Mar 2010 18:58:12 +0000 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> Message-ID: <5c6f2a5d1003251158j850da20ka310d24c1f6be819@mail.gmail.com> On Thu, Mar 25, 2010 at 6:33 PM, Adam Olsen wrote: > IMO, we should go for the simplest of all usage: a NaN singleton, > shared by float and Decimal, compares equal to itself, unsortable, no > payload. I don't think disposing of payloads is much of an option for Decimal: it would break hundreds (literally!) of the decimal testcases, and would mean that the decimal module would no longer comply with the standard it's supposed to be based on. Note that in contrast we're free to alter the meaning of ==, <=, comparisons for the decimal module, since those aren't covered by the standard anyway; someone looking for standards compliance can limit themselves to using Decimal.compare. I much prefer Nick's proposal (in the python-dev bit of this thread: this thread seems to keep moving between python-ideas and python-dev :). > IEEE 754 doesn't say NaN should compare false; it says it should > compare "unordered". Indeed it does. > ?We can't do that as we're not using a 4-way > comparison (which wouldn't generalize to complex anyway), so we have > to make up our own solution. Not necessarily: the standard already gives solutions to the problem of mapping a 4-way comparison to {true, false}. It defines (in section 5.11) a number of predicates that map the four possible relations (LT, GT, EQ, UN) to true and false. Amongst those predicates are 'compareQuietEqual', which maps EQ to True and LT, GT, UN to False (and hence gives nan == nan --> False), and 'compareSignalingEqual', which does the same but also raises a floating-point exception for UN. It doesn't say anything about how those predicates should be spelled in any particular language, though; currently, Python's == corresponds to compareQuietEqual. Unfortunately neither of those predicates gets us round the reflexivity problem. Mark From greg.ewing at canterbury.ac.nz Fri Mar 26 02:25:06 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Mar 2010 14:25:06 +1300 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: <20100325114800.072bfa1a@o> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> <20100325114800.072bfa1a@o> Message-ID: <4BAC0CF2.4090600@canterbury.ac.nz> spir ? wrote: > (Else there should be a distinction between equality assignment and identity assignemt? > b = a # ==> a is b and a == b > b := a # ==> a is b and possibly a == b Eiffel's position on this seems to be that there should be no distinction -- a copy of a value should always compare equal to the original value, regardless of type. Eiffel even seems to extend this to conversions, so that if you convert an int to a float, the resulting float should compare equal to the original int, even if some precision was lost in the conversion. (Incidentally, that's one principle we would be choosing *not* to follow if we decide to compare floats and Decimals based on their exact values.) -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 26 03:01:46 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Mar 2010 15:01:46 +1300 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: <83DAEA2C-05CF-45EC-BCE7-0B30490DDEC1@masklinn.net> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> <83DAEA2C-05CF-45EC-BCE7-0B30490DDEC1@masklinn.net> Message-ID: <4BAC158A.80601@canterbury.ac.nz> Masklinn wrote: > Excuse me but NaNs propagate right? So we'd have not only > float('nan') == float('nan'), but also float('nan') = float('nan') + 1 > right? The fact that one of the operands was a NaN propagates, but it doesn't have to be that particular NaN object. So NaN + 1 could create a new NaN object to return as its result. > [Meyer:] >> It is rather dangerous indeed to depart from the fundamental laws of mathematics. > > The very idea of IEEE-754 floats are a > a definite departure from the fundamental laws of mathematics, They're a departure from the laws of *real numbers*, but Meyer is using the word "mathematics" in a much broader sense here -- meaning any formal system that you can reason about rigorously. If your formal system has a notion of equality but includes values that don't compare equal to themselves, it seriously screws up your ability to reason about it. For Meyer, this is a *big* problem, because making it possible to reason rigorously about programs is something that Eiffel gets really anal about^H^H^H^H^H^H^H^H^H^H ^H^H^H^H^H^H one of the underpinnings of Eiffel's design. The more I think about it, the more I think that the only reason for needing NaNs in the first place is if you don't have, or don't want to use, an exception mechanism. As Meyer points out, treating NaN == NaN as false is not fundamentally any more correct than treating it as true. One interpretation or the other might be appropriate in a particular algorithm, but this needs to be considered on a case-by-case basis. So the *pythonic* way to handle NaNs is not to have them at all, but to raise an exception whenever an operation would produce a NaN. Code that knows what to do can catch the exception and proceed appropriately. Another possibility is to extend the boolean type with a NaB that propagates in the same way as a NaN. But something still has to give somewhere, because what happens when you do this? x = float('nan') y = float('nan') if x == y: ... else: ... If NaN == NaN is a NaB, then neither branch of the 'if' is appopriate, so the only correct thing to do would be to raise an exception when trying to branch on a NaB. So we have three correctness-preserving possibilites: 1) Always raise an exception as soon as a NaN is produced. 2) Propagate NaNs through arithmetic but raise an exception when attempting to compare them. 3) Return a NaB when comparing NaNs, and raise an exception when attempting to branch on a NaB. -- Greg From rhamph at gmail.com Fri Mar 26 03:26:28 2010 From: rhamph at gmail.com (Adam Olsen) Date: Thu, 25 Mar 2010 20:26:28 -0600 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: <4BAC158A.80601@canterbury.ac.nz> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> <83DAEA2C-05CF-45EC-BCE7-0B30490DDEC1@masklinn.net> <4BAC158A.80601@canterbury.ac.nz> Message-ID: On Thu, Mar 25, 2010 at 20:01, Greg Ewing wrote: > So we have three correctness-preserving possibilites: > > 1) Always raise an exception as soon as a NaN is > ? produced. > > 2) Propagate NaNs through arithmetic but raise an > ? exception when attempting to compare them. This forces you to explicitly compare them when desired, but it's even simpler than I realized (and doesn't require new functions). def func(a, b): if not isnan(a) and not isnan(b) and a == b: return 1.0 return math.sin(a*b)**(a-b) All it requires is you check for NaN before your normal comparison. It's even backwards compatible with Python 2.6! If that example is a little verbose for you, just make isnan take multiple arguments and return true if any are NaN. > 3) Return a NaB when comparing NaNs, and raise an > ? exception when attempting to branch on a NaB. -- Adam Olsen, aka Rhamphoryncus From denis.spir at gmail.com Fri Mar 26 11:07:39 2010 From: denis.spir at gmail.com (spir =?UTF-8?B?4pij?=) Date: Fri, 26 Mar 2010 11:07:39 +0100 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: <4BAC0CF2.4090600@canterbury.ac.nz> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> <20100325114800.072bfa1a@o> <4BAC0CF2.4090600@canterbury.ac.nz> Message-ID: <20100326110739.464fa248@o> On Fri, 26 Mar 2010 14:25:06 +1300 Greg Ewing wrote: > Eiffel's position on this seems to be that there should > be no distinction -- a copy of a value should always > compare equal to the original value, regardless of type. Exactly, as I understand Eiffel. And, actually, I tend to support this point of view. > Eiffel even seems to extend this to conversions, so that > if you convert an int to a float, the resulting float should > compare equal to the original int, even if some precision > was lost in the conversion. Yes, there is a post-condition equivalent to "assert(newval==val)". That's why (from this point of view): >>> int(1.2) 1 >>> int(1.7) 1 should not even be possible (--> exception). The programmer has to tell the language about losing information, and thus losing equality, using eg round(). [The same indeed as when converting a huge int to float.] > (Incidentally, that's one principle we would be choosing > *not* to follow if we decide to compare floats and Decimals > based on their exact values.) What do you mean, exactly? There may be a reference type, then the condition is Ref(val)==Ref(newval). Or both type are referent (eg between int and float there may be loss of info in both directions). Denis ________________________________ vit esse estrany ? spir.wikidot.com From denis.spir at gmail.com Sat Mar 27 10:30:03 2010 From: denis.spir at gmail.com (spir =?UTF-8?B?4pij?=) Date: Sat, 27 Mar 2010 10:30:03 +0100 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: <4BAC158A.80601@canterbury.ac.nz> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> <83DAEA2C-05CF-45EC-BCE7-0B30490DDEC1@masklinn.net> <4BAC158A.80601@canterbury.ac.nz> Message-ID: <20100327103003.24db568b@o> On Fri, 26 Mar 2010 15:01:46 +1300 Greg Ewing wrote: > The more I think about it, the more I think that the > only reason for needing NaNs in the first place is if > you don't have, or don't want to use, an exception > mechanism. > > As Meyer points out, treating NaN == NaN as false is > not fundamentally any more correct than treating it > as true. One interpretation or the other might be > appropriate in a particular algorithm, but this needs > to be considered on a case-by-case basis. [In a language or a given case where yielding a NaN does not raise an exception from scratch.] It seems to me there are various comparison cases (esp. in an OO language, case 2): -1- Comparing (any) Nan to non_NaN should return False. a = b = 1 a == b # ==> False -2- Comparing a given NaN with itself should return True. a = b = a a == b # ==> True -3- Comparing various NaNs is the undefined case. Actually, I think it's the only case where a theoretical NaB makes sense. In my opinion, as Greg says, the python way would here be raising an exception. But only in that case. Denis ________________________________ vit esse estrany ? spir.wikidot.com From qrczak at knm.org.pl Sat Mar 27 11:21:17 2010 From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk) Date: Sat, 27 Mar 2010 11:21:17 +0100 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: <4BAC0CF2.4090600@canterbury.ac.nz> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> <20100325114800.072bfa1a@o> <4BAC0CF2.4090600@canterbury.ac.nz> Message-ID: <3f4107911003270321u43638252tc59506a22236d062@mail.gmail.com> 2010/3/26 Greg Ewing : > Eiffel even seems to extend this to conversions, so that > if you convert an int to a float, the resulting float should > compare equal to the original int, even if some precision > was lost in the conversion. Such equality is not transitive (unless 1000000000 and 1000000001 are equal as ints which would be nonsense). The original problem with NaN is a consequence of an unfortunate decision to unify numeric equality with object equivalence. If they were distinguished, their behavior would be obvious: NaN != NaN NaN eq NaN 0.0 == -0.0 0.0 ne -0.0 42 == 42.0 42 ne 42.0 Hash tables would use object equivalence of course. If you have a type for a time which includes the local time and the time zone, and < > <= >= compare which time is earlier, then numeric == should be true for times denoting the same time through different time zones, but they should not be equivalent. Lisp and Scheme distinguish these relations. -- Marcin Kowalczyk From dickinsm at gmail.com Sat Mar 27 13:28:20 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 27 Mar 2010 12:28:20 +0000 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: <3f4107911003270321u43638252tc59506a22236d062@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> <20100325114800.072bfa1a@o> <4BAC0CF2.4090600@canterbury.ac.nz> <3f4107911003270321u43638252tc59506a22236d062@mail.gmail.com> Message-ID: <5c6f2a5d1003270528p38bf1889m2583e2ebf1ea7b6f@mail.gmail.com> On Sat, Mar 27, 2010 at 10:21 AM, Marcin 'Qrczak' Kowalczyk wrote: > The original problem with NaN is a consequence of an unfortunate > decision to unify numeric equality with object equivalence. If they > were distinguished, their behavior would be obvious: Agreed, except that it's not clear to me that this was actually an unfortunate decision. The results in this context are unfortunate, yes, but it could well be that distinguishing numeric equality and object equivalence would add unacceptable complexity to the language for those trying to learn it. Questions about 'is' versus '==' are especially common on the mailing lists, and adding a third equivalence relation wouldn't help newcomers. Mark From dickinsm at gmail.com Sat Mar 27 13:39:13 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 27 Mar 2010 12:39:13 +0000 Subject: [Python-ideas] [Python-Dev] Why is nan != nan? In-Reply-To: <201003271105.28852.steve@pearwood.info> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <201003271105.28852.steve@pearwood.info> Message-ID: <5c6f2a5d1003270539mc6cb47fs6cf8222a077ccfe1@mail.gmail.com> [Moving back to python-ideas] On Sat, Mar 27, 2010 at 12:05 AM, Steven D'Aprano wrote: > Certainly not. That defeats the whole purpose of NANs. I wish floating > point calculations in Python would return NANs rather than raise the > exceptions they do now. I'd prefer the opposite: that arithmetic operations and math functions raise rather than produce nans; this seems friendlier for users who don't want to know or care about infinities and nans. But I agree that there are uses for nonstop mode. How about default behaviour like the above, with a "from __options__ import non-stop-arithmetic" for people who want 1./0. to give infinity and sqrt(-1) to give nan? In order words, default behaviour roughly corresponds to the IEEE 754 spec with the 'invalid operation', 'overflow' and 'divide-by-zero' operations trapped (exactly as the default context in the decimal module does currently), and the non-stop behaviour corresponds to IEEE 754 with none of those operations trapped and returning default values. A more elaborate proposal would allow individual control over the three signals above. (Full control over the 'underflow' and 'inexact' signals isn't really feasible given C's poor support for these things.) . Mark From dickinsm at gmail.com Sat Mar 27 14:22:54 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 27 Mar 2010 13:22:54 +0000 Subject: [Python-ideas] [Python-Dev] Why is nan != nan? In-Reply-To: <5c6f2a5d1003270539mc6cb47fs6cf8222a077ccfe1@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <201003261157.39250.steve@pearwood.info> <20100326011901.6F4133A4080@sparrow.telecommunity.com> <201003271105.28852.steve@pearwood.info> <5c6f2a5d1003270539mc6cb47fs6cf8222a077ccfe1@mail.gmail.com> Message-ID: <5c6f2a5d1003270622w622d172el9f91bd8bf8632aa7@mail.gmail.com> On Sat, Mar 27, 2010 at 12:39 PM, Mark Dickinson wrote: > > > How about default behaviour like the above, with a "from __options__ > import non-stop-arithmetic" for people who want 1./0. to give infinity > and sqrt(-1) to give nan? Hmm. That would be a horrible way to control the option. But the idea's the same: give some way to enable non-stop arithmetic. Some sort of context manager might make sense: with non_stop_arithmetic: result = calculation() if isnan(result): ... Mark From stephen at xemacs.org Sat Mar 27 16:46:57 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 28 Mar 2010 00:46:57 +0900 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: <3f4107911003270321u43638252tc59506a22236d062@mail.gmail.com> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> <20100325114800.072bfa1a@o> <4BAC0CF2.4090600@canterbury.ac.nz> <3f4107911003270321u43638252tc59506a22236d062@mail.gmail.com> Message-ID: <87k4sxg426.fsf@uwakimon.sk.tsukuba.ac.jp> Marcin 'Qrczak' Kowalczyk writes: > The original problem with NaN is a consequence of an unfortunate > decision to unify numeric equality with object equivalence. If they > were distinguished, their behavior would be obvious: > NaN != NaN > NaN eq NaN What do you mean by that? NaN is a class, not an instance! > 0.0 == -0.0 > 0.0 ne -0.0 > 42 == 42.0 > 42 ne 42.0 As for the other examples, I can only sigh, 'Ah, the joys of "general abstract nonsense".'[1] I have some (abstract) sympathy for Mark's proposal for a with_nonstop_arithmetic context manager, but isn't it really a YAGNI for the Python language? (As opposed to high performance modules like numpy.) Footnotes: [1] I refer to Bourbaki's(?) name for category theory. From bruce at leapyear.org Sun Mar 28 00:47:34 2010 From: bruce at leapyear.org (Bruce Leban) Date: Sat, 27 Mar 2010 16:47:34 -0700 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: <87k4sxg426.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> <20100325114800.072bfa1a@o> <4BAC0CF2.4090600@canterbury.ac.nz> <3f4107911003270321u43638252tc59506a22236d062@mail.gmail.com> <87k4sxg426.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: I'm missing something from this discussion. Is this an academic exercise or are there scenarios that don't work well with things as they are? It's hard to determine if a suggested change solve the problem without knowing what the problem is. Here's a problem: I find it annoying that in some (other) languages there's no easy way to trap integer overflow. It's easy to write code that's wrong accidentally. But changing the language definition to fail when an overflow happens would break some valid programs. In python, results that return nan or inf aren't exactly silent failures but they are quiet. (Personally, I would have made them non-hashable so you couldn't accidentally stick one in a dict but that's another problem.) Anyway, of silent/quiet failure is the issue, then what we need is a way to make the failure not silent, e.g., wrap it in a try/except block like this that says, convert any nans to NanErrors: try with NanError: blah except NanError: blah Inside this scope, it might be useful to run code that does allow nans, which means we'd also need something like: without NanError: blah This all seems a bit too magic in how it would work. This is along the line of Mark Dickinson's suggestion, but the default behavior should stay the same as it is now. --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun Mar 28 10:04:08 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 28 Mar 2010 17:04:08 +0900 Subject: [Python-ideas] Why is nan != nan? In-Reply-To: References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com> <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com> <20100325114800.072bfa1a@o> <4BAC0CF2.4090600@canterbury.ac.nz> <3f4107911003270321u43638252tc59506a22236d062@mail.gmail.com> <87k4sxg426.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87iq8gg9dz.fsf@uwakimon.sk.tsukuba.ac.jp> Bruce Leban writes: > (Personally, I would have made them non-hashable so you couldn't > accidentally stick one in a dict but that's another problem.) No, it's not. AIUI, that is one of the main issues that's being discussed in this thread. Most people seem reasonably satisfied with other behavior, so the discussion bounces back and forth between whether NaNs should be hashable (and, perhaps more often, whether they should all be equal or not), and how any change in identity or equality behavior might affect computations in existing programs.