From zuo at chopin.edu.pl  Fri Apr  1 00:39:02 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Fri, 1 Apr 2011 00:39:02 +0200
Subject: [Python-ideas] The namedtuple.abc recipe (was: namedtuple()
	subclasses again)
In-Reply-To: <20110325140637.GB2329@chopin.edu.pl>
References: <20110325140637.GB2329@chopin.edu.pl>
Message-ID: <20110331223902.GA2624@chopin.edu.pl>

PS. For the record, the final recipe is here:

http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/

Bon App?tit!
*j



From tjreedy at udel.edu  Fri Apr  1 02:57:01 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 31 Mar 2011 20:57:01 -0400
Subject: [Python-ideas] Adding a half-float (16-bit) type to PEP 3118
 (and possibly the struct module?)
In-Reply-To: <AANLkTimg8yN4z6qFBOFLEn_ObT0aMHA-T+ZQHtKdPRuL@mail.gmail.com>
References: <AANLkTikmZ=FPo6-SwoKjxYAY8kYsVNAY8YgMDsfnwa4K@mail.gmail.com>	<AANLkTi=bTQXmH+kCkewD9TY0HrW2w+HopEQ0HRc6GCz0@mail.gmail.com>	<AANLkTi=r313w7L8YJLTnyfRitjVR3E6JX-0OpwJ7y3rd@mail.gmail.com>	<AANLkTimwUKnYHkF=AciuJ-LbCfaY8BO_FtsKTjyXRJtG@mail.gmail.com>	<in2aig$pak$1@dough.gmane.org>
	<AANLkTimg8yN4z6qFBOFLEn_ObT0aMHA-T+ZQHtKdPRuL@mail.gmail.com>
Message-ID: <in37sq$rna$2@dough.gmane.org>

On 3/31/2011 1:58 PM, Alexander Belopolsky wrote:
> On Thu, Mar 31, 2011 at 12:36 PM, Robert Kern<robert.kern at gmail.com>  wrote:
> ..
>> The proposed letter code is 'e', as used in numpy. I'm not sure of the logic
>> that went behind the choice, except perhaps that 'e' is near 'd' and 'f'.
>
> So it is  'e' for half,

No, 'e' for 'elfin' ;-).

  'f' for single and 'd' for double.
-- 
Terry Jan Reedy



From wickedgrey at gmail.com  Fri Apr  1 03:09:37 2011
From: wickedgrey at gmail.com (Eli Stevens (Gmail))
Date: Thu, 31 Mar 2011 18:09:37 -0700
Subject: [Python-ideas] Adding a half-float (16-bit) type to PEP 3118
 (and possibly the struct module?)
In-Reply-To: <AANLkTimW-jwLfdk=qun+WL=cK7t1L=g4vbnkEP8_gyTq@mail.gmail.com>
References: <AANLkTikmZ=FPo6-SwoKjxYAY8kYsVNAY8YgMDsfnwa4K@mail.gmail.com>
	<EE877C3D-F856-442F-9B9C-A45A75A46F37@gmail.com>
	<AANLkTimDmcTrZui6gBV0w9H4WBTuE1uoO1Pv7ncU+rEC@mail.gmail.com>
	<AB79954A-7CDC-478F-AE18-D768CAA81B8D@gmail.com>
	<AANLkTimW-jwLfdk=qun+WL=cK7t1L=g4vbnkEP8_gyTq@mail.gmail.com>
Message-ID: <AANLkTi=CRfDFNzynZfu0ByoBQHnC=KVefaRQNYaWjZO_@mail.gmail.com>

On Wed, Mar 30, 2011 at 5:32 PM, Eli Stevens (Gmail)
<wickedgrey at gmail.com> wrote:
> On Wed, Mar 30, 2011 at 5:06 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>> I think the struct module addition for float16 could be handled separately and much more easily (since a half would fit in a double, but a long long double won't).
>
> Okay, if no other objections get raised, I'll open a new issue for it
> (probably sometime tomorrow).

The issue is here:

http://bugs.python.org/issue11734

> Yes, I will try and compile/test CPython and build a patch for
> _struct.c from the current repo.

I am a little unclear as to the purpose of structmember.{h,c}; does
the .h file need a line for the half-float type along the lines of:

#define T_HALFFLOAT	21

And do PyMember_GetOne and PyMember_SetOne need corresponding entries
in their switch statements?  Something like:

    case T_HALFFLOAT:
        // FIXME: need half support
        break;
    case T_FLOAT:
        v = PyFloat_FromDouble((double)*(float*)addr);
        break;

And:

    case T_HALFFLOAT:{
        // FIXME: needs half support
        break;
        }
    case T_FLOAT:{
        double double_val = PyFloat_AsDouble(v);
        if ((double_val == -1) && PyErr_Occurred())
            return -1;
        *(float*)addr = (float)double_val;
        break;
        }

The unit tests I've added for the struct.pack and struck.unpack
functions don't seem to need it, but I want to make sure there isn't
something I'm missing.

Apologies if this should be moved to python-dev; just let me know and
I can repost there.

Thanks!
Eli


From python at mrabarnett.plus.com  Fri Apr  1 03:32:53 2011
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 01 Apr 2011 02:32:53 +0100
Subject: [Python-ideas] Adding a half-float (16-bit) type to PEP 3118
 (and possibly the struct module?)
In-Reply-To: <in37sq$rna$2@dough.gmane.org>
References: <AANLkTikmZ=FPo6-SwoKjxYAY8kYsVNAY8YgMDsfnwa4K@mail.gmail.com>	<AANLkTi=bTQXmH+kCkewD9TY0HrW2w+HopEQ0HRc6GCz0@mail.gmail.com>	<AANLkTi=r313w7L8YJLTnyfRitjVR3E6JX-0OpwJ7y3rd@mail.gmail.com>	<AANLkTimwUKnYHkF=AciuJ-LbCfaY8BO_FtsKTjyXRJtG@mail.gmail.com>	<in2aig$pak$1@dough.gmane.org>	<AANLkTimg8yN4z6qFBOFLEn_ObT0aMHA-T+ZQHtKdPRuL@mail.gmail.com>
	<in37sq$rna$2@dough.gmane.org>
Message-ID: <4D952B45.6050003@mrabarnett.plus.com>

On 01/04/2011 01:57, Terry Reedy wrote:
> On 3/31/2011 1:58 PM, Alexander Belopolsky wrote:
>> On Thu, Mar 31, 2011 at 12:36 PM, Robert Kern<robert.kern at gmail.com>
>> wrote:
>> ..
>>> The proposed letter code is 'e', as used in numpy. I'm not sure of
>>> the logic
>>> that went behind the choice, except perhaps that 'e' is near 'd' and
>>> 'f'.
>>
>> So it is 'e' for half,
>
> No, 'e' for 'elfin' ;-).
>
I thought it was because when you first hear someone talking about
"half-float" you say "eh?" :-)

> 'f' for single and 'd' for double.


From masklinn at masklinn.net  Mon Apr  4 10:31:11 2011
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 4 Apr 2011 10:31:11 +0200
Subject: [Python-ideas] Improving the expressivity of function annotations
Message-ID: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>

While the PEP 3107 annotations are pretty nice, and an interesting improvements re third-party static analysis of Python code, I think they are insufficient in three areas which makes them? insufficient. And even if third-parties solved these issues independently, it likely would not reach criticality.

type parameters (generics)
==========================

Python simply has no syntax (at the moment) to express the idea of generics and generic containers. The simplest way (and one I find syntactically clean, YMMV) would be to define ``__getitem__`` on ``type`` instances (with e.g. ``list[int]`` yielding a list with some kind of ``generics`` attribute), but that is of course unavailable to third-party libraries as they should not override builtins. Plus it would have to be added on ``type`` itself to ensure all types have this property (and I am not sure how C types are defined in relation to ``type``).

The possibility to perform this call would probably need to be opt-in or opt-out, as generics don't make sense for all types (or maybe all types could accept an empty generic spec e.g. ``int[]``? though I'm not sure that is even valid python syntax)

Type parameters greatly increase the expressivity of a type specification and make deep type checks possible, creating much static safety by avoiding implicit or explicit (unsafe) type casts.

sum types
=========

It is not uncommon for Python functions and methods to take or return a value from a set of types rather than from a type itself. That is also (I would guess) one of the reasons why ``isinstance`` can take a tuple of types rather than being limited to a single type (as with Javascript's or Java's ``instanceof``).

As previously, it would probably be easy to alter ``type`` to syntactically support this construct: defining ``__or__`` on ``type`` as yielding some kind of TypeSet (or Types) instance, which would be a set of all OR'ed types (with the right subclass hooks set up so ``isinstance`` works correctly).

It would result in something like:

    Sum = (A | B | C)
    assert issubclass(A, Sum)
    assert issubclass(B, Sum)
    assert issubclass(C, Sum)

Structural types
================

The third issue is also the biggest by far. A small portion of it is resolved by abcs (especially if tools include special support for ABCs), but not all of it by a long shot: statically defining and checking "duck types". abcs check for structure, so they are a convenient label for *some* "duck" types but their meaning isn't even well defined (e.g. does the ``item: Mapping`` signature mean ``item`` implements the abstract methods of Mapping, or does it have to *also* implement the mixin methods, even if Mapping was not actually mixed in it?)

Anyway this is where "structural types" come in: defining a type not by its name but by its shape (a set of methods and properties, and their signatures).

Again, this can be handled by defining (enough) abcs, but it soon becomes tiresome as ABCs are still pretty verbose to write and I don't doubt there would soon be about 5 million versions of ``Readable`` (requirement of a single no-args method ``read`` producing bytes). As a result, the ability to define a type as a set of methods (and their signatures) without having to give it an explicit name would ? I think ? be great.

On the other hand, this is *the* case for which I have no syntactical idea, definitely not a simple one, by simply augmenting existing types with new properties (without introducing actual new syntax). The OCaml syntax ``< method: (*input) -> output; method2: (*input) -> output; property: output >`` definitely does not fit.

Your thoughts?

From cmjohnson.mailinglist at gmail.com  Mon Apr  4 10:44:43 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Sun, 3 Apr 2011 22:44:43 -1000
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
Message-ID: <BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>

On Sun, Apr 3, 2011 at 10:31 PM, Masklinn <masklinn at masklinn.net> wrote:

> Structural types
> ================
>
> The third issue is also the biggest by far. A small portion of it is resolved by abcs (especially if tools include special support for ABCs), but not all of it by a long shot: statically defining and checking "duck types". abcs check for structure, so they are a convenient label for *some* "duck" types but their meaning isn't even well defined (e.g. does the ``item: Mapping`` signature mean ``item`` implements the abstract methods of Mapping, or does it have to *also* implement the mixin methods, even if Mapping was not actually mixed in it?)
>
> Anyway this is where "structural types" come in: defining a type not by its name but by its shape (a set of methods and properties, and their signatures).
>
> Again, this can be handled by defining (enough) abcs, but it soon becomes tiresome as ABCs are still pretty verbose to write and I don't doubt there would soon be about 5 million versions of ``Readable`` (requirement of a single no-args method ``read`` producing bytes). As a result, the ability to define a type as a set of methods (and their signatures) without having to give it an explicit name would ? I think ? be great.

It wouldn't be that hard to make a class decorator that would work like:

@interface
class Duck:
    def quack(): pass
    def walk(): pass

class Mallard:
   def quack():
       print("quack")
   def walk():
       print("waddle")

assert issubclass(Mallard, Duck) #Passes.

Where "interface" is a loose term like Go, and anything with the
relevant methods counts as a subclass. It would probably make sense to
add something like that to the abc module.


From masklinn at masklinn.net  Mon Apr  4 10:49:32 2011
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 4 Apr 2011 10:49:32 +0200
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>
Message-ID: <16BF5680-3240-4F66-A60A-83702157E527@masklinn.net>

On 2011-04-04, at 10:44 , Carl M. Johnson wrote:
> It wouldn't be that hard to make a class decorator that would work like:
> 
> @interface
> class Duck:
>    def quack(): pass
>    def walk(): pass
> 
> class Mallard:
>   def quack():
>       print("quack")
>   def walk():
>       print("waddle")
> 
> assert issubclass(Mallard, Duck) #Passes.
> 
> Where "interface" is a loose term like Go, and anything with the
> relevant methods counts as a subclass. It would probably make sense to
> add something like that to the abc module.

The "issue" is roughly the same as for abcs, in that you still have to give a name (and pre-define a type) for something which may not warrant it, and this naming may even serve to muddy the water some cases when used in type signatures (see Mapping abc example)

Otherwise, sure.

From cmjohnson.mailinglist at gmail.com  Mon Apr  4 11:02:17 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Sun, 3 Apr 2011 23:02:17 -1000
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <16BF5680-3240-4F66-A60A-83702157E527@masklinn.net>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>
	<16BF5680-3240-4F66-A60A-83702157E527@masklinn.net>
Message-ID: <BANLkTinbN4Fzj1daJJQEkBZV6Jvgo3PFYg@mail.gmail.com>

On Sun, Apr 3, 2011 at 10:49 PM, Masklinn <masklinn at masklinn.net> wrote:

> The "issue" is roughly the same as for abcs, in that you still have to give a name (and pre-define a type) for something which may not warrant it, and this naming may even serve to muddy the water some cases when used in type signatures (see Mapping abc example)
>
> Otherwise, sure.

I guess I'm not getting it then. What's the alternative? We can either
a) make up an ABC ourselves b) have a prefilled library of ABCs and
pick one off the shelf or c) not use ABCs at all. In the case that
something doesn't "warrant" an ABC, then just pick c. In fact, some
people believe that we should always pick c to be more Pythonic and
duck type-ish.

Or am I missing an option? I suppose there could be a factory of some
sort to make new ABCs based on existing ABCs already in some sort of
library? Is that what you're proposing?


From cmjohnson.mailinglist at gmail.com  Mon Apr  4 11:09:23 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Sun, 3 Apr 2011 23:09:23 -1000
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <BANLkTinbN4Fzj1daJJQEkBZV6Jvgo3PFYg@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>
	<16BF5680-3240-4F66-A60A-83702157E527@masklinn.net>
	<BANLkTinbN4Fzj1daJJQEkBZV6Jvgo3PFYg@mail.gmail.com>
Message-ID: <BANLkTi=BAYEFbQzHK=mxRWH-4AoaiMrUxQ@mail.gmail.com>

>
> Anyway this is where "structural types" come in: defining a type not by its
> name but by its shape (a set of methods and properties, and their
> signatures).
>


Again, this can be handled by defining (enough) abcs, but it soon becomes
> tiresome as ABCs are still pretty verbose to write and I don't doubt there
> would soon be about 5 million versions of ``Readable`` (requirement of a
> single no-args method ``read`` producing bytes). As a result, the ability to
> define a type as a set of methods (and their signatures) without having to
> give it an explicit name would ? I think ? be great.


OK, reading this again, maybe you mean something more like a Java interface?

@interface
class Writer:
     def write(output: bytes) -> None:
     "Takes in bytes, returns None. Raises Foo on error."
     pass

class DuckWriter(Writer):
    def write(output):
       #do something
        return 1

d = DuckWriter()
d.write("hello") #Raises TypeError, output must be bytes
d.write(b"hello") #Raises TypeError, return type must be NoneType
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110403/ea3e00cc/attachment.html>

From cmjohnson.mailinglist at gmail.com  Mon Apr  4 11:17:46 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Sun, 3 Apr 2011 23:17:46 -1000
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <BANLkTi=BAYEFbQzHK=mxRWH-4AoaiMrUxQ@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>
	<16BF5680-3240-4F66-A60A-83702157E527@masklinn.net>
	<BANLkTinbN4Fzj1daJJQEkBZV6Jvgo3PFYg@mail.gmail.com>
	<BANLkTi=BAYEFbQzHK=mxRWH-4AoaiMrUxQ@mail.gmail.com>
Message-ID: <BANLkTim6shMcacOtu3n4gBJjP39m7XsKkQ@mail.gmail.com>

On Sun, Apr 3, 2011 at 11:09 PM, Carl M. Johnson <
cmjohnson.mailinglist at gmail.com> wrote:


>  OK, reading this again, maybe you mean something more like a Java
> interface?
>

Thinking about it a little more (It's bedtime in my timezone! That's my
excuse!), I think there are two separate things that might be nice to have.
One is an easy way to declare Go-style duck typing interfaces, the other is
an easy way to declare Java-style type checking interfaces. I suppose there
could be two class decorators. Call the first one "interface" and the second
one "typechecking". Interface makes things reply to issubclass with True so
long as they have the same methods and parameters. Typechecking creates a
new class that will raise an error if anything trying to subclass it fails
to use the proper types for input and output.

Do those sounds like useful decorators/metaclasses to have?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110403/e4fb93da/attachment.html>

From masklinn at masklinn.net  Mon Apr  4 11:56:12 2011
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 4 Apr 2011 11:56:12 +0200
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <BANLkTinbN4Fzj1daJJQEkBZV6Jvgo3PFYg@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>
	<16BF5680-3240-4F66-A60A-83702157E527@masklinn.net>
	<BANLkTinbN4Fzj1daJJQEkBZV6Jvgo3PFYg@mail.gmail.com>
Message-ID: <1755005B-F12A-4379-B739-0C8308C341FA@masklinn.net>

On 2011-04-04, at 11:02 , Carl M. Johnson wrote:
> On Sun, Apr 3, 2011 at 10:49 PM, Masklinn <masklinn at masklinn.net> wrote:
> 
>> The "issue" is roughly the same as for abcs, in that you still have to give a name (and pre-define a type) for something which may not warrant it, and this naming may even serve to muddy the water some cases when used in type signatures (see Mapping abc example)
>> 
>> Otherwise, sure.
> 
> I guess I'm not getting it then. What's the alternative? We can either
> a) make up an ABC ourselves b) have a prefilled library of ABCs and
> pick one off the shelf or c) not use ABCs at all. In the case that
> something doesn't "warrant" an ABC, then just pick c. In fact, some
> people believe that we should always pick c to be more Pythonic and
> duck type-ish.
> 
> Or am I missing an option? I suppose there could be a factory of some
> sort to make new ABCs based on existing ABCs already in some sort of
> library? Is that what you're proposing?

The problem of option C and not using ABCs (which is indeed what I'm interested in) is that you then lose the possibility to type the object. If you're not using ABCs and you're not using concrete nominative types (e.g. ``dict``, ``list`` or a custom type) what are you left with?

As far as I can tell, nothing, and that's the hole I think needs plugging in the third part of the initial mail.

On 2011-04-04, at 11:09 , Carl M. Johnson wrote:
> OK, reading this again, maybe you mean something more like a Java interface?
> 
> @interface
> class Writer:
>     def write(output: bytes) -> None:
>     "Takes in bytes, returns None. Raises Foo on error."
>     pass
> 
> class DuckWriter(Writer):
>    def write(output):
>       #do something
>        return 1
> 
> d = DuckWriter()
> d.write("hello") #Raises TypeError, output must be bytes
> d.write(b"hello") #Raises TypeError, return type must be NoneType

Yes in that the signature of the methods would be part of the type, no in that the "link" should be implicit (structural) not explicit (nominative).

Your example does not show anything as it would be a function annotation (not a type annotation), but making it write to a file-like object would work: what I'd like is something along those lines (ignore the notation for now)

class Foo:
    def write_to(stream: < write(bytes) -> None >):
        # stuff

class StringWriter:
    def write(output: str) -> None
        # stuff
class BytesWriter:
    def write(output: bytes) -> None
        # stuff

d = DuckWriter()
d.write_to(StringWriter()) -> TypeError, not a structural subtype of what write_to needs
d.write_to(BytesWriter()) -> OK

On 2011-04-04, at 11:17 , Carl M. Johnson wrote:
> On Sun, Apr 3, 2011 at 11:09 PM, Carl M. Johnson <
> cmjohnson.mailinglist at gmail.com> wrote:
>> OK, reading this again, maybe you mean something more like a Java
>> interface?
> 
> Thinking about it a little more (It's bedtime in my timezone! That's my
> excuse!), I think there are two separate things that might be nice to have.
> One is an easy way to declare Go-style duck typing interfaces, the other is
> an easy way to declare Java-style type checking interfaces. I suppose there
> could be two class decorators. Call the first one "interface" and the second
> one "typechecking". Interface makes things reply to issubclass with True so
> long as they have the same methods and parameters. Typechecking creates a
> new class that will raise an error if anything trying to subclass it fails
> to use the proper types for input and output.
> 
> Do those sounds like useful decorators/metaclasses to have?

Maybe, but that's not the scope I'm considering at all, I'm only talking about function annotations.

From p.f.moore at gmail.com  Mon Apr  4 12:34:51 2011
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 4 Apr 2011 11:34:51 +0100
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <1755005B-F12A-4379-B739-0C8308C341FA@masklinn.net>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>
	<16BF5680-3240-4F66-A60A-83702157E527@masklinn.net>
	<BANLkTinbN4Fzj1daJJQEkBZV6Jvgo3PFYg@mail.gmail.com>
	<1755005B-F12A-4379-B739-0C8308C341FA@masklinn.net>
Message-ID: <BANLkTincN-9-NRUOaGEgvff5_605P5KdnQ@mail.gmail.com>

On 4 April 2011 10:56, Masklinn <masklinn at masklinn.net> wrote:
> Yes in that the signature of the methods would be part of the type, no in that the "link" should be implicit (structural)
> not explicit (nominative).
>
> Your example does not show anything as it would be a function annotation (not a type annotation), but making it write
> to a file-like object would work: what I'd like is something along those lines (ignore the notation for now)
>
> class Foo:
> ? ?def write_to(stream: < write(bytes) -> None >):
> ? ? ? ?# stuff
>
> class StringWriter:
> ? ?def write(output: str) -> None
> ? ? ? ?# stuff
> class BytesWriter:
> ? ?def write(output: bytes) -> None
> ? ? ? ?# stuff
>
> d = DuckWriter()
> d.write_to(StringWriter()) -> TypeError, not a structural subtype of what write_to needs
> d.write_to(BytesWriter()) -> OK

I'm not at all sure I follow your intent or terminology (structural vs
nominative) here - perhaps you could provide a full (but short!)
example of what you're trying to do. The above is close, but names
like "Foo" and hiding actual implementations behind "# stuff" is
confusing me.

If I follow what you're after, surely the #stuff in StringWriter would
naturally raise a TypeError when you tried to use a bytes object as if
it were a string? That's essentially what duck typing is about. If
you're trying to enforce stricter typing (closer to the call site, the
nearest Python can get to "compile time" type checking) then why are
you trying to avoid ABCs? Isn't that what they are for?

If what you are after is not (in some sense) a Pythonic version of
static type checking, then I apologise for misunderstanding, but could
you clarify?

Paul.


From masklinn at masklinn.net  Mon Apr  4 12:58:20 2011
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 4 Apr 2011 12:58:20 +0200
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <BANLkTincN-9-NRUOaGEgvff5_605P5KdnQ@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>
	<16BF5680-3240-4F66-A60A-83702157E527@masklinn.net>
	<BANLkTinbN4Fzj1daJJQEkBZV6Jvgo3PFYg@mail.gmail.com>
	<1755005B-F12A-4379-B739-0C8308C341FA@masklinn.net>
	<BANLkTincN-9-NRUOaGEgvff5_605P5KdnQ@mail.gmail.com>
Message-ID: <7A44F45F-C127-47B9-AD36-4E95E624AFFE@masklinn.net>

On 2011-04-04, at 12:34 , Paul Moore wrote:
> 
> I'm not at all sure I follow your intent or terminology (structural vs
> nominative) here - perhaps you could provide a full (but short!)
> example of what you're trying to do. The above is close, but names
> like "Foo" and hiding actual implementations behind "# stuff" is
> confusing me.
Well thing is I'm hiding the implementation because it's not relevant to my proposal, which is about the function annotations and what they express (actual static checking being supposed to be implemented by third-party tools).

As to structural v nominative:

* Nominative typing is typing based on names, as in Java for instance: a type and a name are the same thing, subtypes explicitly inherit(/extend) their super types by name. In Python, this is what you do when you use isinstance (without using ABCs anyway): you check that an object has a type of a given name.

* Structural typing is basically static duck typing: you ignore names, and instead you check for "shapes". Types are seen as sets of methods (a method being a triplet of a name, an input type and an output type). So instead of checking that an object has the right type-name, you check that it has the right methods. And you can do that statically.

> If I follow what you're after, surely the #stuff in StringWriter would
> naturally raise a TypeError when you tried to use a bytes object as if
> it were a string? That's essentially what duck typing is about.
Yes, what I'm after is giving the opportunity to third-party tools, via an extension to function annotations, to statically check for this.

> If
> you're trying to enforce stricter typing (closer to the call site, the
> nearest Python can get to "compile time" type checking) then why are
> you trying to avoid ABCs? Isn't that what they are for?
Because defining ABCs is verbose, that's cool if you're going to use them quite a lot or as mixins, but defining an ABC for a single type in a single method is verbose and (for short method sets) not as clear.

And as I mentioned there's the issue of the ABC's scope: are mixin methods included in the requirements to be a subclass/instance of ABCs or not? If not, why not? If yes, doesn't that make ABCs much less useful for a huge bunch of objects in the wild?

And third, there's the issue that encoding all subsets of existing concrete types in abcs yields an exponential explosion in cases.

While not all cases are used by a long short consider the mere concept of "file-like object": some will iterate it, some will read() it, some will readline() or readlines() it, should they interact with bytes or with strings, does the file-like object need to support seek() or not, etc? These are all questions which would not be solved easily by a single ABC.

> If what you are after is not (in some sense) a Pythonic version of
> static type checking, then I apologise for misunderstanding, but could
> you clarify?
What I'm after is giving third-party tools the opportunity of structural typing, which is static duck typing.

From p.f.moore at gmail.com  Mon Apr  4 14:48:51 2011
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 4 Apr 2011 13:48:51 +0100
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <7A44F45F-C127-47B9-AD36-4E95E624AFFE@masklinn.net>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>
	<16BF5680-3240-4F66-A60A-83702157E527@masklinn.net>
	<BANLkTinbN4Fzj1daJJQEkBZV6Jvgo3PFYg@mail.gmail.com>
	<1755005B-F12A-4379-B739-0C8308C341FA@masklinn.net>
	<BANLkTincN-9-NRUOaGEgvff5_605P5KdnQ@mail.gmail.com>
	<7A44F45F-C127-47B9-AD36-4E95E624AFFE@masklinn.net>
Message-ID: <BANLkTim8jmC7QQpt6ocXg6QLEy1a9ON=cA@mail.gmail.com>

On 4 April 2011 11:58, Masklinn <masklinn at masklinn.net> wrote:
>> If what you are after is not (in some sense) a Pythonic version of
>> static type checking, then I apologise for misunderstanding, but could
>> you clarify?
> What I'm after is giving third-party tools the opportunity of structural typing, which is static duck typing.

OK. I understand now, thanks for the clarification.

I'm -1 on this. Static typing is not appropriate for Python, IMO.

Just as one example, note that even ABCs are not statically
deterministic. Consider ABCMeta.register and ABCMeta.__subclasshook__.
If a tool implements any form of static type checking, it will by
definition fail to handle a certain proportion of valid Python
programs. Whether this is acceptable is, of course, a personal
decision.

Paul.


From ncoghlan at gmail.com  Mon Apr  4 15:06:23 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 4 Apr 2011 23:06:23 +1000
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
Message-ID: <BANLkTimEadK_u2d9qJdVEtNLkz8eyh54DQ@mail.gmail.com>

If what you want is static typing, perhaps using a statically typed
language would be a good place to start?

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From masklinn at masklinn.net  Mon Apr  4 15:33:08 2011
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 4 Apr 2011 15:33:08 +0200
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <BANLkTim8jmC7QQpt6ocXg6QLEy1a9ON=cA@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>
	<16BF5680-3240-4F66-A60A-83702157E527@masklinn.net>
	<BANLkTinbN4Fzj1daJJQEkBZV6Jvgo3PFYg@mail.gmail.com>
	<1755005B-F12A-4379-B739-0C8308C341FA@masklinn.net>
	<BANLkTincN-9-NRUOaGEgvff5_605P5KdnQ@mail.gmail.com>
	<7A44F45F-C127-47B9-AD36-4E95E624AFFE@masklinn.net>
	<BANLkTim8jmC7QQpt6ocXg6QLEy1a9ON=cA@mail.gmail.com>
Message-ID: <772B6CF6-900A-4778-817A-5DC64477F968@masklinn.net>

On 2011-04-04, at 14:48 , Paul Moore wrote:
> 
> Just as one example, note that even ABCs are not statically
> deterministic. Consider ABCMeta.register and ABCMeta.__subclasshook__.
ABCs can be interpreted as structural type tags, making ABCMeta.register and ABCMeta.__subclasshook__ ignorable at the cost of potential false positives (a type with the right structure would pass the static type check even if it didn't pass an issubclass check). I'm more than willing to live with that. 

> If a tool implements any form of static type checking, it will by
> definition fail to handle a certain proportion of valid Python
> programs.
Sure, static checking tools will not be able to handle some properties of Python program, but it's not like that's a new issue (there are already static checking tools for Python and they already deal with these issues via heuristics and white/black lists of members, and there is quite a history of dynamically typed languages with optional/partial/progressive static typing), and that doesn't prevent trying to get the best bang from our buck re. optional/progressive static annotations.

And the vast majority of static type systems have loopholes, Python just has far more than others.

On 2011-04-04, at 15:06 , Nick Coghlan wrote:
> If what you want is static typing, perhaps using a statically typed
> language would be a good place to start?
Mostly what I was interested in was making function annotations useful. I'm guessing they were introduced to be used yet due to the limitations I outlined their actual value falls like a rock as soon as you're using anything more complex than strings and numbers, especially if you tend to use "duck" types a lot.



From ncoghlan at gmail.com  Mon Apr  4 16:08:03 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 5 Apr 2011 00:08:03 +1000
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <772B6CF6-900A-4778-817A-5DC64477F968@masklinn.net>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<BANLkTiksTtm4B7w7fZULpSwqFuOyzk3fEg@mail.gmail.com>
	<16BF5680-3240-4F66-A60A-83702157E527@masklinn.net>
	<BANLkTinbN4Fzj1daJJQEkBZV6Jvgo3PFYg@mail.gmail.com>
	<1755005B-F12A-4379-B739-0C8308C341FA@masklinn.net>
	<BANLkTincN-9-NRUOaGEgvff5_605P5KdnQ@mail.gmail.com>
	<7A44F45F-C127-47B9-AD36-4E95E624AFFE@masklinn.net>
	<BANLkTim8jmC7QQpt6ocXg6QLEy1a9ON=cA@mail.gmail.com>
	<772B6CF6-900A-4778-817A-5DC64477F968@masklinn.net>
Message-ID: <BANLkTinzKKHCaEZqp8mUJkdahGDjkyJvNQ@mail.gmail.com>

On Mon, Apr 4, 2011 at 11:33 PM, Masklinn <masklinn at masklinn.net> wrote:
> On 2011-04-04, at 15:06 , Nick Coghlan wrote:
>> If what you want is static typing, perhaps using a statically typed
>> language would be a good place to start?
> Mostly what I was interested in was making function annotations useful. I'm guessing they were introduced to be used yet due to the limitations I outlined their actual value falls like a rock as soon as you're using anything more complex than strings and numbers, especially if you tend to use "duck" types a lot.

They're not useful for comprehensive static typing without defining
additional conventions. But that's OK, since that's not their purpose
- at the language level, they're just a place to associate arbitrary
data with parameters and the function as a whole, typically for use by
specific decorators that are also applied at function definition time.
Which annotations you use and which decorators you use will
accordingly be task specific. (For example, here's a simple one that
uses function annotations to define relatively type safe calls out to
C functions via ctypes:
http://code.activestate.com/recipes/576731-c-function-decorator/)

As it stands, there's absolutely nothing stopping you implementing the
suggestions in your original post as an external library (albeit
without being able to use infix syntax on the builtin types, but using
tuples instead should work just fine for the sum case, and you can
define your own Generic ABCs for the generics case). As far as the
structural types go, just use function annotations and appropriate
decorators on the methods in an ABC definition (although you would
want your own new metaclass to enforce the additional rules).

As far as the language itself goes, however, we made a deliberate
decision not to ascribe any specific semantics to function
annotations, and nothing has happened in the intervening time to
convince us to go back on that decision.

It would require an annotations-based approach to a task to prove
incredibly popular for us to bless for it inclusion in the standard
library. If anything was going to happen on that front, my money would
actually be on some utilities in ctypes, such as the cookbook recipe I
linked above. This is *not* an area where theoretical arguments are
going to hold any water: only practical solutions that help make code
dealing with real-world problems significantly more readable.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From dickinsm at gmail.com  Mon Apr  4 17:28:11 2011
From: dickinsm at gmail.com (Mark Dickinson)
Date: Mon, 4 Apr 2011 16:28:11 +0100
Subject: [Python-ideas] Adding a half-float (16-bit) type to PEP 3118
 (and possibly the struct module?)
In-Reply-To: <in04kj$gqg$1@dough.gmane.org>
References: <AANLkTikmZ=FPo6-SwoKjxYAY8kYsVNAY8YgMDsfnwa4K@mail.gmail.com>
	<AANLkTim6=AYmuvPn4Pna7jZSC3jZOXtixHKVGN5TsfV5@mail.gmail.com>
	<in01oi$te0$1@dough.gmane.org>
	<AANLkTinG0QE__NwaPNPsTCYo2We8fR=OO+mtfeSu1qUa@mail.gmail.com>
	<in04kj$gqg$1@dough.gmane.org>
Message-ID: <BANLkTi=9-TFLR33MmsiyOk4ek=A-YY1uaw@mail.gmail.com>

On Wed, Mar 30, 2011 at 9:42 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On 3/30/11 3:05 PM, Mark Dickinson wrote:
>> [OT]: How is NumPy's float16 type implemented? ?Is it clever enough to
>> do correct rounding for all basic arithmetic operations, or does it
>> suffer from the double-rounding problems that you'd get from (convert
>> operands to float64; do op in float64; round back to float16)?
>
> We do the latter, I'm afraid. Except with float32 instead of float64.

[Still OT] Confession time:  after asking this question, I had a
sneaking suspicion that it was a stupid one.  And having had time to
think a bit, it turns out that it is.  As far as I can tell, assuming
round-half-to-even, there *are* no double rounding problems for
primitive arithmetic operations with the (convert operands to float32;
do operation in float32; convert back).  This is probably a well-known
fact in numerics land, and I feel embarrassed for not noticing.  The
key point is that the precision of float32 (24 bit precision) is at
least double that of float16 (11 bit precision), plus a couple of
extra bits;  it's then easy to see that there can be no problems with
multiplication, a mite harder to see that addition and subtraction are
fine, and just a tiny bit harder again to show that division of two
float16s can never give a result that'll be rounded the wrong way
under the double (to float32, then to float16) rounding.

Sheepishly,

Mark


From tjreedy at udel.edu  Mon Apr  4 20:22:08 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 04 Apr 2011 14:22:08 -0400
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
Message-ID: <ind28g$csd$1@dough.gmane.org>

I am not sure, what, if anything, we should do to make annotations more 
useful. I think it is too soon to do anything *official* yet.

On 4/4/2011 4:31 AM, Masklinn wrote:

> type parameters (generics)

I do not know what 'generics' means (and neither, that I could find, 
does Wikipedia) but type-parameter is not obvious as a synonym ;-).

> ``type`` instances (with e.g. ``list[int]`` yielding a list with some
> kind of ``generics`` attribute),

That sounds a lot like array with a typecode. You can use a dict 
subclass or UserDict to get a mapping with restricted keys and values, 
with bad inputs ignored, coerced, or error raising as you please. Ditto 
for lists.

> Sum = (A | B | C)
 > assert issubclass(A, Sum)

That appears to be exactly the same as
Sum = (A,B,C)
assert issubclass(A, Sum)

Sum2=Sum|D # same as
Sum2=Sum+(D,)

so the purpose of the duplication is not obvious.

> Anyway this is where "structural types" come in: defining a type not
> by its name but by its shape (a set of methods and properties, and
> their signatures).

When it comes to tracebacks, multiple unbound functions with duplicate 
f.__name__ == '<lambda>' attributes are inferior to functions with, in a 
particular context, unique names. I would feel the same about multiple 
stypes with .__name__ == '<stype>'.

-- 
Terry Jan Reedy



From guido at python.org  Mon Apr  4 20:25:27 2011
From: guido at python.org (Guido van Rossum)
Date: Mon, 4 Apr 2011 11:25:27 -0700
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <ind28g$csd$1@dough.gmane.org>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
Message-ID: <BANLkTinr_+a5m0zW2TBh01qYpUae4urpDA@mail.gmail.com>

On Mon, Apr 4, 2011 at 11:22 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> I am not sure, what, if anything, we should do to make annotations more
> useful. I think it is too soon to do anything *official* yet.

For sure it is too soon -- the intention was that the new syntax would
enable 3rd party experiments. We're still waiting for reports on such
experiments.

-- 
--Guido van Rossum (python.org/~guido)


From ianb at colorstudy.com  Mon Apr  4 20:42:29 2011
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 4 Apr 2011 13:42:29 -0500
Subject: [Python-ideas] Dict view on list of two-tuples
Message-ID: <BANLkTimd4mO-0sS-FjaACmsHzH62nGEqyw@mail.gmail.com>

It would be nice if the stdlib had a data structure that represented a dict
view on a list of key/value tuples.  This has come up a bit when thinking
about common web request/response objects, as they all have an object like
this but they all have slightly different APIs (and sometimes slightly
different notions of what they want to do).

For instance:

d = multidict([('a', '1'), ('b', '2'), ('a', '3')])

There are some open questions:

d['a'] -- '1' or '3'?
d.items() -- could be [('a', '1'), ('b', '2'), ('a', '3')], or [('b', '2'),
('a', '3')]
how do you get all values for 'a'?  (e.g., d.getall('a') == ['1', '3'])
what does "del d['a']" do?
what does "d['a'] = '4'" do?  It may overwrite or add a value.  Both are
valid -- how do you do the other.
d.pop() etc need specifying.
does d represent a view or a copy of the list?  I.e., are updates to the
list reflected in d?

Some implementations only represent ordering of individual values, e.g.,
they'll show that '3' comes after '1'.  But I need a complete ordering, that
is, I need to know the key ordering is 'a', 'b', 'a' (it does not
necessarily need to be shown through d.items(), but some method should
expose this).  Data structures with an internal representation like {'a':
['1', '3'], 'b': ['2']} are not sufficient.

For reference, my own opinion of an implementation:
https://bitbucket.org/ianb/webob/src/tip/webob/multidict.py#cl-13

  Ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110404/b1589aed/attachment.html>

From fuzzyman at gmail.com  Mon Apr  4 20:47:09 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Mon, 4 Apr 2011 19:47:09 +0100
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <ind28g$csd$1@dough.gmane.org>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
Message-ID: <BANLkTim7MpzB_BK4X4kkrKUzoTVHmWk86Q@mail.gmail.com>

On 4 April 2011 19:22, Terry Reedy <tjreedy at udel.edu> wrote:

> I am not sure, what, if anything, we should do to make annotations more
> useful. I think it is too soon to do anything *official* yet.
>
> On 4/4/2011 4:31 AM, Masklinn wrote:
>
>  type parameters (generics)
>>
>
> I do not know what 'generics' means (and neither, that I could find, does
> Wikipedia) but type-parameter is not obvious as a synonym ;-).
>
>

http://en.wikipedia.org/wiki/Generic_programming

It's a standard term for languages like C# and Java, but if you don't use
these languages there is no reason you should know it. Generics is a
solution (hack - but still an elegant hack) that allows you to write
"generic" containers and functions that can work with any types whilst still
being type safe. For example, in C# you might have a list type that can
contain elements of a specific type. The list can be written once using
generics, then can be used with any type - with the specific type being
specified (the type parameter) at compile time where you use it. E.g.

    var list_of_ints = new List[int]();

HTH,

Michael Foord


>
>  ``type`` instances (with e.g. ``list[int]`` yielding a list with some
>> kind of ``generics`` attribute),
>>
>
> That sounds a lot like array with a typecode. You can use a dict subclass
> or UserDict to get a mapping with restricted keys and values, with bad
> inputs ignored, coerced, or error raising as you please. Ditto for lists.
>
>
>  Sum = (A | B | C)
>>
> > assert issubclass(A, Sum)
>
> That appears to be exactly the same as
>
> Sum = (A,B,C)
> assert issubclass(A, Sum)
>
> Sum2=Sum|D # same as
> Sum2=Sum+(D,)
>
> so the purpose of the duplication is not obvious.
>
>
>  Anyway this is where "structural types" come in: defining a type not
>> by its name but by its shape (a set of methods and properties, and
>> their signatures).
>>
>
> When it comes to tracebacks, multiple unbound functions with duplicate
> f.__name__ == '<lambda>' attributes are inferior to functions with, in a
> particular context, unique names. I would feel the same about multiple
> stypes with .__name__ == '<stype>'.
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110404/d7ba5da5/attachment.html>

From masklinn at masklinn.net  Mon Apr  4 21:04:55 2011
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 4 Apr 2011 21:04:55 +0200
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <BANLkTim7MpzB_BK4X4kkrKUzoTVHmWk86Q@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
	<BANLkTim7MpzB_BK4X4kkrKUzoTVHmWk86Q@mail.gmail.com>
Message-ID: <A0DF094C-10EE-48B1-B767-4CD2DB25AE51@masklinn.net>

On 2011-04-04, at 20:47 , Michael Foord wrote:
> 
> It's a standard term for languages like C# and Java, but if you don't use
> these languages there is no reason you should know it. Generics is a
> solution (hack - but still an elegant hack) that allows you to write
> "generic" containers and functions that can work with any types whilst still
> being type safe.

Why do you find generics to be a hack?



From fuzzyman at gmail.com  Mon Apr  4 21:05:56 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Mon, 4 Apr 2011 20:05:56 +0100
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <A0DF094C-10EE-48B1-B767-4CD2DB25AE51@masklinn.net>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
	<BANLkTim7MpzB_BK4X4kkrKUzoTVHmWk86Q@mail.gmail.com>
	<A0DF094C-10EE-48B1-B767-4CD2DB25AE51@masklinn.net>
Message-ID: <BANLkTinT=gKR1JDxtX=-cN3MXzjMdqzCvg@mail.gmail.com>

On 4 April 2011 20:04, Masklinn <masklinn at masklinn.net> wrote:

> On 2011-04-04, at 20:47 , Michael Foord wrote:
> >
> > It's a standard term for languages like C# and Java, but if you don't use
> > these languages there is no reason you should know it. Generics is a
> > solution (hack - but still an elegant hack) that allows you to write
> > "generic" containers and functions that can work with any types whilst
> still
> > being type safe.
>
> Why do you find generics to be a hack?
>
>
Because with a good dynamic type system they are completely unnecessary.

Michael

-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110404/14a2f2b2/attachment.html>

From masklinn at masklinn.net  Mon Apr  4 21:03:33 2011
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 4 Apr 2011 21:03:33 +0200
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <ind28g$csd$1@dough.gmane.org>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
Message-ID: <362D9EBA-9688-43A8-84A4-DF393386CD73@masklinn.net>

On 2011-04-04, at 20:22 , Terry Reedy wrote:
> 
>> ``type`` instances (with e.g. ``list[int]`` yielding a list with some
>> kind of ``generics`` attribute),
> That sounds a lot like array with a typecode.
It's similar, but extended to any arbitrary type, including non-collection types.

> You can use a dict subclass or UserDict to get a mapping with restricted keys and values, with bad inputs ignored, coerced, or error raising as you please. Ditto for lists.
That is not very useful for function annotations, especially when using built-in collections. Users would be restricted to non-built-in types and static systems still would not have any information on what could and could not go into e.g. collections.

>> Sum = (A | B | C)
> > assert issubclass(A, Sum)
> 
> That appears to be exactly the same as
> Sum = (A,B,C)
> assert issubclass(A, Sum)
> 
> Sum2=Sum|D # same as
> Sum2=Sum+(D,)
> 
> so the purpose of the duplication is not obvious.
Again, the issue is function annotations, which was my context. The example was just to give an idea of the operation performed. In function annotations, unless you went with type parameters and ``tuple[A, B, C]``, ``(A, B, C)`` would likely be interpreted as "a triple of instances of types A, B and C".

>> Anyway this is where "structural types" come in: defining a type not
>> by its name but by its shape (a set of methods and properties, and
>> their signatures).
> When it comes to tracebacks, multiple unbound functions with duplicate f.__name__ == '<lambda>' attributes are inferior to functions with, in a particular context, unique names. I would feel the same about multiple stypes with .__name__ == '<stype>'.
I'm not sure what you're getting at here: structural types are static features, the displayed types would be the original concrete types you created, there would be no loss of runtime information compared to current behaviors.

From masklinn at masklinn.net  Mon Apr  4 21:30:52 2011
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 4 Apr 2011 21:30:52 +0200
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <BANLkTinT=gKR1JDxtX=-cN3MXzjMdqzCvg@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
	<BANLkTim7MpzB_BK4X4kkrKUzoTVHmWk86Q@mail.gmail.com>
	<A0DF094C-10EE-48B1-B767-4CD2DB25AE51@masklinn.net>
	<BANLkTinT=gKR1JDxtX=-cN3MXzjMdqzCvg@mail.gmail.com>
Message-ID: <B09DA325-367F-46D0-B2CE-3B6F50E5BB74@masklinn.net>

On 2011-04-04, at 21:05 , Michael Foord wrote:
> On 4 April 2011 20:04, Masklinn <masklinn at masklinn.net> wrote:
>> On 2011-04-04, at 20:47 , Michael Foord wrote:
>>> It's a standard term for languages like C# and Java, but if you don't use
>>> these languages there is no reason you should know it. Generics is a
>>> solution (hack - but still an elegant hack) that allows you to write
>>> "generic" containers and functions that can work with any types whilst
>> still
>>> being type safe.
>> 
>> Why do you find generics to be a hack?
>> 
> Because with a good dynamic type system they are completely unnecessary.

If you go that way, types themselves are unnecessary "and therefore hacks", static or not.

I don't think that makes much sense, though I can see you were probably replying in jest I was interested in the answer.

From fuzzyman at gmail.com  Mon Apr  4 21:47:45 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Mon, 4 Apr 2011 20:47:45 +0100
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <B09DA325-367F-46D0-B2CE-3B6F50E5BB74@masklinn.net>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
	<BANLkTim7MpzB_BK4X4kkrKUzoTVHmWk86Q@mail.gmail.com>
	<A0DF094C-10EE-48B1-B767-4CD2DB25AE51@masklinn.net>
	<BANLkTinT=gKR1JDxtX=-cN3MXzjMdqzCvg@mail.gmail.com>
	<B09DA325-367F-46D0-B2CE-3B6F50E5BB74@masklinn.net>
Message-ID: <BANLkTik4dxGxyGSqWm=khQh4C9+F9jX-fw@mail.gmail.com>

On 4 April 2011 20:30, Masklinn <masklinn at masklinn.net> wrote:

> On 2011-04-04, at 21:05 , Michael Foord wrote:
> > On 4 April 2011 20:04, Masklinn <masklinn at masklinn.net> wrote:
> >> On 2011-04-04, at 20:47 , Michael Foord wrote:
> >>> It's a standard term for languages like C# and Java, but if you don't
> use
> >>> these languages there is no reason you should know it. Generics is a
> >>> solution (hack - but still an elegant hack) that allows you to write
> >>> "generic" containers and functions that can work with any types whilst
> >> still
> >>> being type safe.
> >>
> >> Why do you find generics to be a hack?
> >>
> > Because with a good dynamic type system they are completely unnecessary.
>
> If you go that way, types themselves are unnecessary "and therefore hacks",
> static or not.
>
> I don't think that makes much sense, though I can see you were probably
> replying in jest I was interested in the answer.


I wasn't entirely joking, and no a dynamic type system doesn't make types
themselves redundant - just the declaration of them all over the place (and
often multiple times for the same use in languages like Java and C#).
Generics are a hack within the language syntax to tell the compiler that
different types *might* be used (and allow you to refer to these types in
your implementation without knowing what they will be), whereas a smarter
compiler could deduce this for itself anyway.

All the best,

Michael

-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110404/7147d94a/attachment.html>

From mwm at mired.org  Mon Apr  4 22:33:46 2011
From: mwm at mired.org (Mike Meyer)
Date: Mon, 4 Apr 2011 16:33:46 -0400
Subject: [Python-ideas] Improving the expressivity of function
 annotations (getting off-topic?)
In-Reply-To: <BANLkTik4dxGxyGSqWm=khQh4C9+F9jX-fw@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
	<BANLkTim7MpzB_BK4X4kkrKUzoTVHmWk86Q@mail.gmail.com>
	<A0DF094C-10EE-48B1-B767-4CD2DB25AE51@masklinn.net>
	<BANLkTinT=gKR1JDxtX=-cN3MXzjMdqzCvg@mail.gmail.com>
	<B09DA325-367F-46D0-B2CE-3B6F50E5BB74@masklinn.net>
	<BANLkTik4dxGxyGSqWm=khQh4C9+F9jX-fw@mail.gmail.com>
Message-ID: <20110404163346.18b85c17@bhuda.mired.org>

On Mon, 4 Apr 2011 20:47:45 +0100
Michael Foord <fuzzyman at gmail.com> wrote:
> I wasn't entirely joking, and no a dynamic type system doesn't make types
> themselves redundant - just the declaration of them all over the place (and
> often multiple times for the same use in languages like Java and C#).

A good implementation of static types will do that for you as well.

> Generics are a hack within the language syntax to tell the compiler that
> different types *might* be used (and allow you to refer to these types in
> your implementation without knowing what they will be), whereas a smarter
> compiler could deduce this for itself anyway.

Generics (in the named languages) feel like a hack because, well, they
were hacked onto a type system (inherited from C) that didn't allow
for such things. If you design your type system ab initio expecting
that you're going to allow different types to appear in some places -
so long as they satisfy the appropriate conditions - then what you
have may look like generics, but they aren't such hacks. It doesn't
matter whether the type system is dynamic (conditions checked at run
time) or static (conditions checked at compile time).

     <mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/consulting.html
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org


From greg.ewing at canterbury.ac.nz  Mon Apr  4 23:49:47 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 05 Apr 2011 09:49:47 +1200
Subject: [Python-ideas] Improving the expressivity of
	function	annotations
In-Reply-To: <BANLkTik4dxGxyGSqWm=khQh4C9+F9jX-fw@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
	<BANLkTim7MpzB_BK4X4kkrKUzoTVHmWk86Q@mail.gmail.com>
	<A0DF094C-10EE-48B1-B767-4CD2DB25AE51@masklinn.net>
	<BANLkTinT=gKR1JDxtX=-cN3MXzjMdqzCvg@mail.gmail.com>
	<B09DA325-367F-46D0-B2CE-3B6F50E5BB74@masklinn.net>
	<BANLkTik4dxGxyGSqWm=khQh4C9+F9jX-fw@mail.gmail.com>
Message-ID: <4D9A3CFB.3020807@canterbury.ac.nz>

Michael Foord wrote:
> 
> Generics are a hack within the language syntax to tell the 
> compiler that different types *might* be used (and allow you to refer to 
> these types in your implementation without knowing what they will be), 
> whereas a smarter compiler could deduce this for itself anyway.

You seem to be conflating the idea of generic types, aka
parametric types, with the particular syntax used to express
them in certain languages. Some of the syntaxes do seem
rather hackish, particularly in C++. But I don't think the
concept itself is hackish at all.

-- 
Greg



From steve at pearwood.info  Tue Apr  5 00:36:08 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 05 Apr 2011 08:36:08 +1000
Subject: [Python-ideas] Improving the expressivity of
	function	annotations
In-Reply-To: <BANLkTimEadK_u2d9qJdVEtNLkz8eyh54DQ@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<BANLkTimEadK_u2d9qJdVEtNLkz8eyh54DQ@mail.gmail.com>
Message-ID: <4D9A47D8.6030205@pearwood.info>

Nick Coghlan wrote:
> If what you want is static typing, perhaps using a statically typed
> language would be a good place to start?


You might be interested in Cobra, which is a Python-influenced language 
with optional static typing.

http://cobra-language.com/

Comparison to Python:
http://cobra-language.com/docs/python/



-- 
Steven


From steve at pearwood.info  Tue Apr  5 00:56:24 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 05 Apr 2011 08:56:24 +1000
Subject: [Python-ideas] Dict view on list of two-tuples
In-Reply-To: <BANLkTimd4mO-0sS-FjaACmsHzH62nGEqyw@mail.gmail.com>
References: <BANLkTimd4mO-0sS-FjaACmsHzH62nGEqyw@mail.gmail.com>
Message-ID: <4D9A4C98.5050200@pearwood.info>

Ian Bicking wrote:
> It would be nice if the stdlib had a data structure that represented a dict
> view on a list of key/value tuples.  This has come up a bit when thinking
> about common web request/response objects, as they all have an object like
> this but they all have slightly different APIs (and sometimes slightly
> different notions of what they want to do).

I don't know whether "dict view" is the right terminology, but I've 
frequently found myself writing code that accepts a mapping, either a 
dict or a list of (key,value) items. So far I've handled this in an ad 
hoc way, only implementing as much functionality as I've needed at the 
time. I can't say it's been a major burden.

Perhaps the simplest thing would be a key/value mapping (kvmap?) type 
that wraps such a list with a dict interface. Which parts of the dict 
interface (all of it? only some of it?) remains to be answered.

For sure though, it would have to loosen the requirement that keys are 
unique and unsorted. (key,value) pairs are only useful for when you need 
non-unique, ordered keys, otherwise I'd just use a dict :)


> Data structures with an internal representation like {'a':
> ['1', '3'], 'b': ['2']} are not sufficient.

How do you know? The ordered dict implementation in the standard library 
has an internal dict representation (or at least it did, when it was on 
ActiveState). Why do you care whether the implementation uses a dict 
internally, so long as the interface matches the specified (and as yet 
not entirely decided) API?



-- 
Steven



From ianb at colorstudy.com  Tue Apr  5 01:51:38 2011
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 4 Apr 2011 18:51:38 -0500
Subject: [Python-ideas] Dict view on list of two-tuples
In-Reply-To: <4D9A4C98.5050200@pearwood.info>
References: <BANLkTimd4mO-0sS-FjaACmsHzH62nGEqyw@mail.gmail.com>
	<4D9A4C98.5050200@pearwood.info>
Message-ID: <BANLkTin5ijsLSY=75GzfHyxaUFx75_B-EQ@mail.gmail.com>

On Mon, Apr 4, 2011 at 5:56 PM, Steven D'Aprano <steve at pearwood.info> wrote:

> Data structures with an internal representation like {'a':
>> ['1', '3'], 'b': ['2']} are not sufficient.
>>
>
> How do you know? The ordered dict implementation in the standard library
> has an internal dict representation (or at least it did, when it was on
> ActiveState). Why do you care whether the implementation uses a dict
> internally, so long as the interface matches the specified (and as yet not
> entirely decided) API?


Well, I can say at least that it would not be sufficient for me to use it in
WebOb -- I wouldn't consider it acceptable to lose the full ordering of all
keys, as would happen if you used this structure.  It might be suitable for
some other use, but I have no such use in mind at the moment, so diverging
from this it would no longer be the idea that led me to start the thread ;)

  Ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110404/995f3a10/attachment.html>

From steve at pearwood.info  Tue Apr  5 02:14:49 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 05 Apr 2011 10:14:49 +1000
Subject: [Python-ideas] Dict view on list of two-tuples
In-Reply-To: <BANLkTin5ijsLSY=75GzfHyxaUFx75_B-EQ@mail.gmail.com>
References: <BANLkTimd4mO-0sS-FjaACmsHzH62nGEqyw@mail.gmail.com>
	<4D9A4C98.5050200@pearwood.info>
	<BANLkTin5ijsLSY=75GzfHyxaUFx75_B-EQ@mail.gmail.com>
Message-ID: <4D9A5EF9.6020608@pearwood.info>

Ian Bicking wrote:
> On Mon, Apr 4, 2011 at 5:56 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> 
>> Data structures with an internal representation like {'a':
>>> ['1', '3'], 'b': ['2']} are not sufficient.
>>>
>> How do you know? The ordered dict implementation in the standard library
>> has an internal dict representation (or at least it did, when it was on
>> ActiveState). Why do you care whether the implementation uses a dict
>> internally, so long as the interface matches the specified (and as yet not
>> entirely decided) API?
> 
> 
> Well, I can say at least that it would not be sufficient for me to use it in
> WebOb -- I wouldn't consider it acceptable to lose the full ordering of all
> keys, as would happen if you used this structure.  It might be suitable for
> some other use, but I have no such use in mind at the moment, so diverging
> from this it would no longer be the idea that led me to start the thread ;)

You have missed my point. What makes you think that using a dict as part 
of the *internal implementation* would mean that you lose the full 
ordering of keys? ordereddict has full ordering of keys, and it uses a 
dict as the internal implementation.

The point is that you shouldn't declare what *implementation* a solution 
may or may not use, only the desired functionality.

Functional requirements: *what* the code should do
Implementation: *how* the code performs the required functions

In my experience as a manager at an IT company, one of the hardest 
lessons for technical people to learn is not to mix implementation 
details in the functional requirements. That and not spending sixteen 
hours solving a problem when the customer has only authorised three :)

You shouldn't rule out implementations based on preconceptions of what 
is or isn't possible. If somebody had declared "ordereddict must not use 
a dict under the hood", we wouldn't have an ordereddict in the standard 
library, or we'd have a sub-standard one.



-- 
Steven



From ianb at colorstudy.com  Tue Apr  5 02:21:25 2011
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 4 Apr 2011 19:21:25 -0500
Subject: [Python-ideas] Dict view on list of two-tuples
In-Reply-To: <4D9A5EF9.6020608@pearwood.info>
References: <BANLkTimd4mO-0sS-FjaACmsHzH62nGEqyw@mail.gmail.com>
	<4D9A4C98.5050200@pearwood.info>
	<BANLkTin5ijsLSY=75GzfHyxaUFx75_B-EQ@mail.gmail.com>
	<4D9A5EF9.6020608@pearwood.info>
Message-ID: <BANLkTimeurNsxKbx--xDkAPQ=7KFTVCRww@mail.gmail.com>

On Mon, Apr 4, 2011 at 7:14 PM, Steven D'Aprano <steve at pearwood.info> wrote:

> Ian Bicking wrote:
>
>> On Mon, Apr 4, 2011 at 5:56 PM, Steven D'Aprano <steve at pearwood.info>
>> wrote:
>>
>>  Data structures with an internal representation like {'a':
>>>
>>>> ['1', '3'], 'b': ['2']} are not sufficient.
>>>>
>>>>  How do you know? The ordered dict implementation in the standard
>>> library
>>> has an internal dict representation (or at least it did, when it was on
>>> ActiveState). Why do you care whether the implementation uses a dict
>>> internally, so long as the interface matches the specified (and as yet
>>> not
>>> entirely decided) API?
>>>
>>
>>
>> Well, I can say at least that it would not be sufficient for me to use it
>> in
>> WebOb -- I wouldn't consider it acceptable to lose the full ordering of
>> all
>> keys, as would happen if you used this structure.  It might be suitable
>> for
>> some other use, but I have no such use in mind at the moment, so diverging
>> from this it would no longer be the idea that led me to start the thread
>> ;)
>>
>
> You have missed my point. What makes you think that using a dict as part of
> the *internal implementation* would mean that you lose the full ordering of
> keys? ordereddict has full ordering of keys, and it uses a dict as the
> internal implementation.
>

I was referring to the representation as a shorthand for what I considered a
common but poor implementation of the concept -- *that specific* internal
representation has specific results which I consider unacceptable (that is,
it keeps the relative ordering of the values of one key, but doesn't keep
the ordering between keys).  Another representation that uses a dict could
be fine, but I didn't offer a generic representation, I offered a specific
one.

  Ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110404/1cba8563/attachment.html>

From cmjohnson.mailinglist at gmail.com  Tue Apr  5 02:44:16 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Mon, 4 Apr 2011 14:44:16 -1000
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <362D9EBA-9688-43A8-84A4-DF393386CD73@masklinn.net>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
	<362D9EBA-9688-43A8-84A4-DF393386CD73@masklinn.net>
Message-ID: <BANLkTikSUPXrZ5hKSmBbQmKtiN8_sR+6VQ@mail.gmail.com>

On Mon, Apr 4, 2011 at 9:03 AM, Masklinn <masklinn at masklinn.net> wrote:

> Again, the issue is function annotations, which was my context. The example was just to give an idea of the operation performed. In function annotations, unless you went with type parameters and ``tuple[A, B, C]``, ``(A, B, C)`` would likely be interpreted as "a triple of instances of types A, B and C".

Yes, but since function annotation don't do anything at present (and
maybe never will by default), your function will have to use a
decorator to turn on the type checking. At that point, it's just a
matter of API design how best to do it. For example, let's say I have
a sum function, and I want it to take an iterator containing some type
and then return something of the same type. In semi-C-ish style, we'd
call this list[int] -> int or list[Generic number] -> Generic number.
But since we're making up our own API for type checking, in this case
we can make it work however we want. We don't need to have a
particular grammar baked into the language. Try something like:

from type_checking_annotations import *

g = Generic()

@typechecked
def sum(iterable: Container(g)) -> g:
    iterator = iter(iterable)
    total = next(iterator)
    for item in iterator:
        total += item
    return total

sum([0, 0.0]) #Raise TypeError: First item in container was type int,
subsequent item was type float


From jsbueno at python.org.br  Tue Apr  5 06:00:26 2011
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Tue, 5 Apr 2011 01:00:26 -0300
Subject: [Python-ideas] Dict view on list of two-tuples
In-Reply-To: <BANLkTimeurNsxKbx--xDkAPQ=7KFTVCRww@mail.gmail.com>
References: <BANLkTimd4mO-0sS-FjaACmsHzH62nGEqyw@mail.gmail.com>
	<4D9A4C98.5050200@pearwood.info>
	<BANLkTin5ijsLSY=75GzfHyxaUFx75_B-EQ@mail.gmail.com>
	<4D9A5EF9.6020608@pearwood.info>
	<BANLkTimeurNsxKbx--xDkAPQ=7KFTVCRww@mail.gmail.com>
Message-ID: <BANLkTik+83c1QTDEni4hgtPKW50bDpTLbA@mail.gmail.com>

On Mon, Apr 4, 2011 at 9:21 PM, Ian Bicking <ianb at colorstudy.com> wrote:
> On Mon, Apr 4, 2011 at 7:14 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> Ian Bicking wrote:
>>>
>>> On Mon, Apr 4, 2011 at 5:56 PM, Steven D'Aprano <steve at pearwood.info>
>>> wrote:
>>>
>>>> Data structures with an internal representation like {'a':
>>>>> ['1', '3'], 'b': ['2']} are not sufficient.

Nonetheless, I can't see as a "dict-view" for this data could
replicate the full order - and be usable
in the sense of using keys.

I think that if one needs the full order as stated, the obvious way to
consume that data is as a sequence
of 2-tuples.

On the other hand, I am +1 for a standardization of this, and for the
implementation of a "dict-view" to an object that otherwise is a
sequence of 2-tuples.

   js
 -><-


From masklinn at masklinn.net  Tue Apr  5 08:27:05 2011
From: masklinn at masklinn.net (Masklinn)
Date: Tue, 5 Apr 2011 08:27:05 +0200
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <BANLkTik4dxGxyGSqWm=khQh4C9+F9jX-fw@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
	<BANLkTim7MpzB_BK4X4kkrKUzoTVHmWk86Q@mail.gmail.com>
	<A0DF094C-10EE-48B1-B767-4CD2DB25AE51@masklinn.net>
	<BANLkTinT=gKR1JDxtX=-cN3MXzjMdqzCvg@mail.gmail.com>
	<B09DA325-367F-46D0-B2CE-3B6F50E5BB74@masklinn.net>
	<BANLkTik4dxGxyGSqWm=khQh4C9+F9jX-fw@mail.gmail.com>
Message-ID: <321091BA-2102-4EBD-B8A8-7A8B1C53D0FF@masklinn.net>

On 2011-04-04, at 21:47 , Michael Foord wrote:
> On 4 April 2011 20:30, Masklinn <masklinn at masklinn.net> wrote:
>> On 2011-04-04, at 21:05 , Michael Foord wrote:
>>> On 4 April 2011 20:04, Masklinn <masklinn at masklinn.net> wrote:
>>>> On 2011-04-04, at 20:47 , Michael Foord wrote:
>>>>> It's a standard term for languages like C# and Java, but if you don't
>> use
>>>>> these languages there is no reason you should know it. Generics is a
>>>>> solution (hack - but still an elegant hack) that allows you to write
>>>>> "generic" containers and functions that can work with any types whilst
>>>> still
>>>>> being type safe.
>>>> 
>>>> Why do you find generics to be a hack?
>>>> 
>>> Because with a good dynamic type system they are completely unnecessary.
>> 
>> If you go that way, types themselves are unnecessary "and therefore hacks",
>> static or not.
>> 
>> I don't think that makes much sense, though I can see you were probably
>> replying in jest I was interested in the answer.
> 
> I wasn't entirely joking, and no a dynamic type system doesn't make types
> themselves redundant
You stated generics (and static types) were unnecessary. As Church demonstrated in 1936 types themselves are unnecessary.

> just the declaration of them all over the place (and
> often multiple times for the same use in languages like Java and C#).
> Generics are a hack within the language syntax to tell the compiler that
> different types *might* be used (and allow you to refer to these types in
> your implementation without knowing what they will be)
That has nothing to do with types themselves, generic or not. Haskell, ML and many other languages have parametric types and generics (potentially in extensions to the core language) as well as type inference (local and global).

If you mean that Java and C#'s suck I have no problem with that, but I still don't see how that ends up yielding "generics are unnecessary". Even if types are inferred, they're still there and still parametric.

> whereas a smarter compiler could deduce this for itself anyway.
I don't think that isn't possible (for all cases) for a non-total language unless the compiler can solve the halting problem. But I might be mistaken.

From masklinn at masklinn.net  Tue Apr  5 08:35:18 2011
From: masklinn at masklinn.net (Masklinn)
Date: Tue, 5 Apr 2011 08:35:18 +0200
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <BANLkTikSUPXrZ5hKSmBbQmKtiN8_sR+6VQ@mail.gmail.com>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
	<362D9EBA-9688-43A8-84A4-DF393386CD73@masklinn.net>
	<BANLkTikSUPXrZ5hKSmBbQmKtiN8_sR+6VQ@mail.gmail.com>
Message-ID: <C791189C-0B25-4A51-B5B1-576194069F4E@masklinn.net>

On 2011-04-05, at 02:44 , Carl M. Johnson wrote:
> On Mon, Apr 4, 2011 at 9:03 AM, Masklinn <masklinn at masklinn.net> wrote:
>> Again, the issue is function annotations, which was my context. The example was just to give an idea of the operation performed. In function annotations, unless you went with type parameters and ``tuple[A, B, C]``, ``(A, B, C)`` would likely be interpreted as "a triple of instances of types A, B and C".
> 
> Yes, but since function annotation don't do anything at present (and
> maybe never will by default), your function will have to use a
> decorator to turn on the type checking. At that point, it's just a
> matter of API design how best to do it. For example, let's say I have
> a sum function, and I want it to take an iterator containing some type
> and then return something of the same type. In semi-C-ish style, we'd
> call this list[int] -> int or list[Generic number] -> Generic number.
> But since we're making up our own API for type checking, in this case
> we can make it work however we want. We don't need to have a
> particular grammar baked into the language.
I would still think that, for the usage of function annotation to take off (and be used by multiple tools), there needs to be some kind of commonality between all tools. At least for the features closest to core.

Indeed, function annotations do currently provide a common (very bare) starting point which was found good enough (at least as an experiment) to replace/supersede/complement the thousand of possible docstring styles of function annotations (from annotations-like to Sphinx info-fields through doxygen, epydoc and dozens of ad-hoc schemes).

The root of my proposal is that, after using both statically typed and dynamically typed languages and writing a bunch of API docs (mostly for Python and Javascript), I don't think the current function annotations provide enough ground for language users to stand on, be it as documentation or as meta-information for static checking tools.

From ncoghlan at gmail.com  Tue Apr  5 12:22:12 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 5 Apr 2011 20:22:12 +1000
Subject: [Python-ideas] Adding a half-float (16-bit) type to PEP 3118
 (and possibly the struct module?)
In-Reply-To: <BANLkTi=9-TFLR33MmsiyOk4ek=A-YY1uaw@mail.gmail.com>
References: <AANLkTikmZ=FPo6-SwoKjxYAY8kYsVNAY8YgMDsfnwa4K@mail.gmail.com>
	<AANLkTim6=AYmuvPn4Pna7jZSC3jZOXtixHKVGN5TsfV5@mail.gmail.com>
	<in01oi$te0$1@dough.gmane.org>
	<AANLkTinG0QE__NwaPNPsTCYo2We8fR=OO+mtfeSu1qUa@mail.gmail.com>
	<in04kj$gqg$1@dough.gmane.org>
	<BANLkTi=9-TFLR33MmsiyOk4ek=A-YY1uaw@mail.gmail.com>
Message-ID: <BANLkTinKRL4HR+2BWYzTyvT83gd7AjbkrA@mail.gmail.com>

On Tue, Apr 5, 2011 at 1:28 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> [Still OT] Confession time: ?after asking this question, I had a
> sneaking suspicion that it was a stupid one.

>From watching yourself and others work on it over the years, I've come
to the conclusion that there are *no* stupid questions when it comes
to float rounding.

If anyone was looking for a field absolutely rife with answers that
are "simple, obvious and dead wrong", avoiding cumulative errors in
binary float manipulation would have to be a prime candidate.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Tue Apr  5 12:34:04 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 5 Apr 2011 20:34:04 +1000
Subject: [Python-ideas] Improving the expressivity of function
	annotations
In-Reply-To: <C791189C-0B25-4A51-B5B1-576194069F4E@masklinn.net>
References: <25A89688-23E0-4F27-829B-128D4F6B67B4@masklinn.net>
	<ind28g$csd$1@dough.gmane.org>
	<362D9EBA-9688-43A8-84A4-DF393386CD73@masklinn.net>
	<BANLkTikSUPXrZ5hKSmBbQmKtiN8_sR+6VQ@mail.gmail.com>
	<C791189C-0B25-4A51-B5B1-576194069F4E@masklinn.net>
Message-ID: <BANLkTimJEpgJ79CURBZMxP-D0JsBrq8Uxw@mail.gmail.com>

On Tue, Apr 5, 2011 at 4:35 PM, Masklinn <masklinn at masklinn.net> wrote:
> I would still think that, for the usage of function annotation to take off (and be used by multiple tools), there needs to be some kind of commonality between all tools. At least for the features closest to core.

You're free to think that. We disagree, as is explicitly documented in
PEP 3107: "Function annotations are nothing more than a way of
associating arbitrary Python expressions with various parts of a
function at compile-time."

Function annotations, on their own, mean absolutely nothing. They only
acquire meaning when associated with a specific consumer of those
annotations. If a decorator takes a lot of arguments about how to
handle particular parameters (or a function's return value), then it
is a prime candidate for refactoring to be annotation based instead.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From cool-rr at cool-rr.com  Tue Apr  5 23:16:57 2011
From: cool-rr at cool-rr.com (cool-RR)
Date: Tue, 5 Apr 2011 23:16:57 +0200
Subject: [Python-ideas] A meaningful `if counter:`
Message-ID: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>

Hello folks,

I noticed today that bool(Counter({'a': 0'})) is True.

Is this wise? I want to be able to do:

    if my_counter:
        whatever

To check whether my counter has any elements. Currently this seems to be
impossible because of this behavior.

Will we have to keep this weird behavior because of backwards compatibility?
If so, perhaps `.elements` could be turned into a smart object so we could
at least do `if my_counter.elements():` and get the expected result.

To people who use `Counter`: Will you find the ability to do boolean checks
on a counter useful?

If you want a patch let me know and I'll write one.


Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110405/e55178b6/attachment.html>

From bruce at leapyear.org  Tue Apr  5 23:34:45 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Tue, 5 Apr 2011 14:34:45 -0700
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
Message-ID: <BANLkTiki0F-wnGpB383_frsHae60R0k8+A@mail.gmail.com>

According to the documentation
http://docs.python.org/release/3.1.3/library/collections.html#collections.Counter.elements
c.elements() returns an empty iterator if there are no elements with counts
> 0.

When you say you want it to be smart, what do you mean besides that?

--- Bruce
*New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April
Fools!
*New!** *Blog post:
http://www.vroospeak.com/2011/04/march-gets-more-madness-next-year.html April
Fools!



On Tue, Apr 5, 2011 at 2:16 PM, cool-RR <cool-rr at cool-rr.com> wrote:

> Hello folks,
>
> I noticed today that bool(Counter({'a': 0'})) is True.
>
> Is this wise? I want to be able to do:
>
>     if my_counter:
>         whatever
>
> To check whether my counter has any elements. Currently this seems to be
> impossible because of this behavior.
>
> Will we have to keep this weird behavior because of backwards
> compatibility? If so, perhaps `.elements` could be turned into a smart
> object so we could at least do `if my_counter.elements():` and get the
> expected result.
>
> To people who use `Counter`: Will you find the ability to do boolean checks
> on a counter useful?
>
> If you want a patch let me know and I'll write one.
>
>
> Ram.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110405/b84e76e8/attachment.html>

From cool-rr at cool-rr.com  Tue Apr  5 23:46:01 2011
From: cool-rr at cool-rr.com (cool-RR)
Date: Tue, 5 Apr 2011 23:46:01 +0200
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <BANLkTiki0F-wnGpB383_frsHae60R0k8+A@mail.gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
	<BANLkTiki0F-wnGpB383_frsHae60R0k8+A@mail.gmail.com>
Message-ID: <BANLkTinD+w_isBMcK82yLQKOfrWhXEBvMA@mail.gmail.com>

On Tue, Apr 5, 2011 at 11:34 PM, Bruce Leban <bruce at leapyear.org> wrote:

> According to the documentation
>
> http://docs.python.org/release/3.1.3/library/collections.html#collections.Counter.elements
> c.elements() returns an empty iterator if there are no elements with counts
> > 0.
>

I am aware of the current behavior.


> When you say you want it to be smart, what do you mean besides that?
>

I mean that it will be like `dict.keys`; an object which behaves similarly
to a list but is implemented more smartly. e.g., `len(counter.elements())`
would be implemented as `sum(counter.values())` for better O complexity, and
`counter.elements().__bool__` would be (trivially) implemented. (You
currently can't do a meaningful `if counter.elements()`).


Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110405/69aaa488/attachment.html>

From python at mrabarnett.plus.com  Tue Apr  5 23:58:21 2011
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 05 Apr 2011 22:58:21 +0100
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <BANLkTinD+w_isBMcK82yLQKOfrWhXEBvMA@mail.gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>	<BANLkTiki0F-wnGpB383_frsHae60R0k8+A@mail.gmail.com>
	<BANLkTinD+w_isBMcK82yLQKOfrWhXEBvMA@mail.gmail.com>
Message-ID: <4D9B907D.1030501@mrabarnett.plus.com>

On 05/04/2011 22:46, cool-RR wrote:
> On Tue, Apr 5, 2011 at 11:34 PM, Bruce Leban <bruce at leapyear.org
> <mailto:bruce at leapyear.org>> wrote:
>
>     According to the documentation
>     http://docs.python.org/release/3.1.3/library/collections.html#collections.Counter.elements
>     c.elements() returns an empty iterator if there are no elements with
>     counts > 0.
>
>
> I am aware of the current behavior.
>
>     When you say you want it to be smart, what do you mean besides that?
>
>
> I mean that it will be like `dict.keys`; an object which behaves
> similarly to a list but is implemented more smartly. e.g.,
> `len(counter.elements())` would be implemented as
> `sum(counter.values())` for better O complexity, and
> `counter.elements().__bool__` would be (trivially) implemented. (You
> currently can't do a meaningful `if counter.elements()`).
>
A count in a Counter can be negative, so `len(counter.elements())`
isn't the same as `sum(counter.values())` (counter.elements() returns
only those elements having a positive count).


From bruce at leapyear.org  Wed Apr  6 00:05:24 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Tue, 5 Apr 2011 15:05:24 -0700
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <BANLkTinD+w_isBMcK82yLQKOfrWhXEBvMA@mail.gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
	<BANLkTiki0F-wnGpB383_frsHae60R0k8+A@mail.gmail.com>
	<BANLkTinD+w_isBMcK82yLQKOfrWhXEBvMA@mail.gmail.com>
Message-ID: <BANLkTikvbg42T7AkYspWQni0emunNoaVLg@mail.gmail.com>

You're mixing implementation details with behavior.

I agree it would be nice if bool(counter.elements()) was False if the list
of elements is empty. Supporting len(counter.elements()) could be done if it
was useful, but I'm not sure what the use case is.

--- Bruce
*New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April
Fools!
*New!** *Blog post:
http://www.vroospeak.com/2011/04/march-gets-more-madness-next-year.html April
Fools!



On Tue, Apr 5, 2011 at 2:46 PM, cool-RR <cool-rr at cool-rr.com> wrote:

> On Tue, Apr 5, 2011 at 11:34 PM, Bruce Leban <bruce at leapyear.org> wrote:
>
>> According to the documentation
>>
>> http://docs.python.org/release/3.1.3/library/collections.html#collections.Counter.elements
>> c.elements() returns an empty iterator if there are no elements with
>> counts > 0.
>>
>
> I am aware of the current behavior.
>
>
>> When you say you want it to be smart, what do you mean besides that?
>>
>
> I mean that it will be like `dict.keys`; an object which behaves similarly
> to a list but is implemented more smartly. e.g., `len(counter.elements())`
> would be implemented as `sum(counter.values())` for better O complexity, and
> `counter.elements().__bool__` would be (trivially) implemented. (You
> currently can't do a meaningful `if counter.elements()`).
>
>
> Ram.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110405/1ee1ba97/attachment.html>

From raymond.hettinger at gmail.com  Wed Apr  6 01:54:15 2011
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 5 Apr 2011 16:54:15 -0700
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
Message-ID: <FF51BDBC-F20A-4884-9DAA-1C309A43CEDA@gmail.com>


On Apr 5, 2011, at 2:16 PM, cool-RR wrote:

> Hello folks,
> 
> I noticed today that bool(Counter({'a': 0'})) is True.
> 
> Is this wise? I want to be able to do:
> 
>     if my_counter:
>         whatever

As explained on tracker where you made the same request,
that counter is not an empty container.  Accordingly, it's bool
value is True.  The python language has been consistent
with this for a long time (as I found-out years ago when
I proposed putting a __len__ attribute on various iterools
when the output iterator length was known, but Guido
wanted the iterator to always have a bool value of False).

A Counter object is essentially just a dictionary with a
__missing__ method to supply an implied zero default count.
Accordingly, it needs to behave as much like a dictionary
as possible.


Raymond


From dbaker3448 at gmail.com  Wed Apr  6 07:31:18 2011
From: dbaker3448 at gmail.com (Dan Baker)
Date: Wed, 6 Apr 2011 00:31:18 -0500
Subject: [Python-ideas] Extending properties to sequence-like behavior
Message-ID: <BANLkTi=qK3NcL=3rBJZwHP+yYdnhyjZ0TQ@mail.gmail.com>

I know the property() built-in allows you to override the behavior of
attribute access in a class to call arbitrary functions where the
syntax looks like a basic assign or lookup. This is handy when you're
importing an external API and want to make it look more Pythonic. Is
there a simple way to extend this so that item lookup or assignment
can be similarly overridden? For instance, if the object I'm importing
(say, a drop-down box in a GUI toolkit) has methods called
GetAllItems, SetAllItems, GetItemAtPos, and SetItemAtPos, I want to be
able to translate:

return box.Items -> return box.GetAllItems()
box.Items = itemlist -> box.SetAllItems(itemlist)
return box.Items[idx] -> return box.GetItemAtPos(idx)
box.Items[idx] = new_item -> box.SetItemAtPos(idx, new_item)

Just using property() doesn't quite work; it handles the "set all" and
"get all" operations easily enough, and "get item" works for any
interface that it would be sane to attempt this on by just getting the
appropriate slice of the list returned by "get all" (although this may
be inefficient compared to calling the real "get item" function), but
"set item" doesn't work because it calls "get all" and then assigns to
a slot in the list returned by that function.

The best solution I've come up with so far is to create an object (of
a class I'll call ListProp) and have the "get" function of the
property return this object. Then I can override the __getitem__ and
__setitem__ methods to call the "get item" and "set item" functions.
The "get all" function no longer works exactly as above, though;
box.Items would instead return the ListProp object. This can be solved
by using __call__ on the ListProp as the "get all" method; then the
only change is to use box.Items() for the "get all" function.

I've attached some scratch code here (Python 2.7) demonstrating the
basic idea. Is there a better way to do this?

Dan Baker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: listprop.py
Type: application/octet-stream
Size: 1036 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110406/46a7ead3/attachment.obj>

From raymond.hettinger at gmail.com  Wed Apr  6 07:52:23 2011
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 5 Apr 2011 22:52:23 -0700
Subject: [Python-ideas] Extending properties to sequence-like behavior
In-Reply-To: <BANLkTi=qK3NcL=3rBJZwHP+yYdnhyjZ0TQ@mail.gmail.com>
References: <BANLkTi=qK3NcL=3rBJZwHP+yYdnhyjZ0TQ@mail.gmail.com>
Message-ID: <6BCFCB04-AF4F-4E88-AFF9-9FD49617DD94@gmail.com>


On Apr 5, 2011, at 10:31 PM, Dan Baker wrote:
> I want to be
> able to translate:
> 
> return box.Items -> return box.GetAllItems()
> box.Items = itemlist -> box.SetAllItems(itemlist)
> return box.Items[idx] -> return box.GetItemAtPos(idx)
> box.Items[idx] = new_item -> box.SetItemAtPos(idx, new_item)
> 
> Just using property() doesn't quite work; it handles the "set all" and
> "get all" operations easily enough, and "get item" works for any
> interface that it would be sane to attempt this on by just getting the
> appropriate slice of the list returned by "get all" (although this may
> be inefficient compared to calling the real "get item" function), but
> "set item" doesn't work because it calls "get all" and then assigns to
> a slot in the list returned by that function.
> 
> The best solution I've come up with so far is to create an object (of
> a class I'll call ListProp) and have the "get" function of the
> property return this object. Then I can override the __getitem__ and
> __setitem__ methods to call the "get item" and "set item" functions.
> The "get all" function no longer works exactly as above, though;
> box.Items would instead return the ListProp object. This can be solved
> by using __call__ on the ListProp as the "get all" method; then the
> only change is to use box.Items() for the "get all" function.
> 
> I've attached some scratch code here (Python 2.7) demonstrating the
> basic idea. Is there a better way to do this?

I don't see any other way.  You're solution is probably the only
workable approach.

Since the [idx] call occurs after the box.Items attribute lookup, 
the box.Items lookup needs to return some object that can a 
subsequent [idx] call using __getitem__ or __setitem__:

>>> from dis import dis
>>> dis(compile('box.Items[idx] = new_item', '', 'exec'))
  1           0 LOAD_NAME                0 (new_item)
              3 LOAD_NAME                1 (box)
              6 LOAD_ATTR                2 (Items)
              9 LOAD_NAME                3 (idx)
             12 STORE_SUBSCR        
             13 LOAD_CONST               0 (None)
             16 RETURN_VALUE        

Thanks for the interesting post.  Nice work in figuring this all out.


Raymond







From digitalxero at gmail.com  Wed Apr  6 08:48:12 2011
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Wed, 6 Apr 2011 02:48:12 -0400
Subject: [Python-ideas] Extending properties to sequence-like behavior
In-Reply-To: <6BCFCB04-AF4F-4E88-AFF9-9FD49617DD94@gmail.com>
References: <BANLkTi=qK3NcL=3rBJZwHP+yYdnhyjZ0TQ@mail.gmail.com>
	<6BCFCB04-AF4F-4E88-AFF9-9FD49617DD94@gmail.com>
Message-ID: <BANLkTinM4MFZg_dqcnYvef5_gVoOKANqWw@mail.gmail.com>

> On Apr 5, 2011, at 10:31 PM, Dan Baker wrote:
>> I want to be
>> able to translate:
>>
>> return box.Items -> return box.GetAllItems()
>> box.Items = itemlist -> box.SetAllItems(itemlist)
>> return box.Items[idx] -> return box.GetItemAtPos(idx)
>> box.Items[idx] = new_item -> box.SetItemAtPos(idx, new_item)

I would translate it to

box.items -> box.GetAllItems()
box.items = itemlist -> box.SetAllItems(itemlist)
box.item[idx] -> box.GetItemAtPos(idx)
box.item[idx] = new_item -> box.SetItemAtPos(idx, new_item)

http://dpaste.com/529172/

Yes you have 2 properties now but it is fairly easy to remember that
items is always getting or setting all of them and item is always
getting or setting a single one


From cool-rr at cool-rr.com  Wed Apr  6 08:59:41 2011
From: cool-rr at cool-rr.com (cool-RR)
Date: Wed, 6 Apr 2011 08:59:41 +0200
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <FF51BDBC-F20A-4884-9DAA-1C309A43CEDA@gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
	<FF51BDBC-F20A-4884-9DAA-1C309A43CEDA@gmail.com>
Message-ID: <BANLkTi=MwOUv2dXoWpoVwQt05zvamE6LOA@mail.gmail.com>

On Wed, Apr 6, 2011 at 1:54 AM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

>
> On Apr 5, 2011, at 2:16 PM, cool-RR wrote:
>
> > Hello folks,
> >
> > I noticed today that bool(Counter({'a': 0'})) is True.
> >
> > Is this wise? I want to be able to do:
> >
> >     if my_counter:
> >         whatever
>
> As explained on tracker where you made the same request,
> that counter is not an empty container.  Accordingly, it's bool
> value is True.  The python language has been consistent
> with this for a long time (as I found-out years ago when
> I proposed putting a __len__ attribute on various iterools
> when the output iterator length was known, but Guido
> wanted the iterator to always have a bool value of False).
>
> A Counter object is essentially just a dictionary with a
> __missing__ method to supply an implied zero default count.
> Accordingly, it needs to behave as much like a dictionary
> as possible.
>
>
> Raymond
>

Okay, this argument applies against `Counter.__bool__`, but not against
`Counter.elements().__bool__`. So I want to hear whether people would find a
 `Counter.elements().__bool__` method useful.


Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110406/e1e38144/attachment.html>

From raymond.hettinger at gmail.com  Wed Apr  6 09:57:43 2011
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 6 Apr 2011 00:57:43 -0700
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <BANLkTi=MwOUv2dXoWpoVwQt05zvamE6LOA@mail.gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
	<FF51BDBC-F20A-4884-9DAA-1C309A43CEDA@gmail.com>
	<BANLkTi=MwOUv2dXoWpoVwQt05zvamE6LOA@mail.gmail.com>
Message-ID: <AA4D0E90-2E55-4627-AAED-8668E59AD9C9@gmail.com>


On Apr 5, 2011, at 11:59 PM, cool-RR wrote:

> Okay, this argument applies against `Counter.__bool__`, but not against `Counter.elements().__bool__`. So I want to hear whether people would find a  `Counter.elements().__bool__` method useful.
> 



if any(c.elements()):
    ...


Raymond



From cool-rr at cool-rr.com  Wed Apr  6 10:00:58 2011
From: cool-rr at cool-rr.com (cool-RR)
Date: Wed, 6 Apr 2011 10:00:58 +0200
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <AA4D0E90-2E55-4627-AAED-8668E59AD9C9@gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
	<FF51BDBC-F20A-4884-9DAA-1C309A43CEDA@gmail.com>
	<BANLkTi=MwOUv2dXoWpoVwQt05zvamE6LOA@mail.gmail.com>
	<AA4D0E90-2E55-4627-AAED-8668E59AD9C9@gmail.com>
Message-ID: <BANLkTincU+qFDvwp926qkDV+D8h3NTruXA@mail.gmail.com>

On Wed, Apr 6, 2011 at 9:57 AM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

>
> On Apr 5, 2011, at 11:59 PM, cool-RR wrote:
>
> > Okay, this argument applies against `Counter.__bool__`, but not against
> `Counter.elements().__bool__`. So I want to hear whether people would find a
>  `Counter.elements().__bool__` method useful.
> >
>
>
>
> if any(c.elements()):
>    ...
>
>
> Raymond
>


Cool, this will be good enough.


Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110406/8d95537f/attachment.html>

From ncoghlan at gmail.com  Wed Apr  6 15:49:08 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 6 Apr 2011 23:49:08 +1000
Subject: [Python-ideas] Extending properties to sequence-like behavior
In-Reply-To: <BANLkTi=qK3NcL=3rBJZwHP+yYdnhyjZ0TQ@mail.gmail.com>
References: <BANLkTi=qK3NcL=3rBJZwHP+yYdnhyjZ0TQ@mail.gmail.com>
Message-ID: <BANLkTi=N_06gM2M0qoGS9xxVwthyxjc3jw@mail.gmail.com>

On Wed, Apr 6, 2011 at 3:31 PM, Dan Baker <dbaker3448 at gmail.com> wrote:
> I've attached some scratch code here (Python 2.7) demonstrating the
> basic idea. Is there a better way to do this?

Perhaps rename the returned class to SequenceProp and have it inherit
from collections.MutableSequence?

Then you could override methods and use GetAllItems/SetAllItems at
appropriate points (e.g. in __iter__, or when the item access methods
are passed slices with start, stop and step all set to None).

It will be a little more work up front, but it's the only way you're
going to get full control over the way client code interacts with your
external sequence objects.

One of the problems you have at the moment is that other mutating
methods of lists (such as .sort() and .reverse()) still aren't going
to work as would be expected if "box.Items" truly was a list. Avoiding
that claim and instead making it clear you only provide the smaller
sequence ABC would help make that clear (and MutableSequence
automatically defines many of those operations in terms of the core
abstract methods, anyway).


Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From bruce at leapyear.org  Wed Apr  6 15:57:50 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Wed, 6 Apr 2011 06:57:50 -0700
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <BANLkTincU+qFDvwp926qkDV+D8h3NTruXA@mail.gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
	<FF51BDBC-F20A-4884-9DAA-1C309A43CEDA@gmail.com>
	<BANLkTi=MwOUv2dXoWpoVwQt05zvamE6LOA@mail.gmail.com>
	<AA4D0E90-2E55-4627-AAED-8668E59AD9C9@gmail.com>
	<BANLkTincU+qFDvwp926qkDV+D8h3NTruXA@mail.gmail.com>
Message-ID: <BANLkTi=vxyXnmoNuaE04z-s1hy7GR5E54Q@mail.gmail.com>

No it won't.

>>> c = Counter()
>>> c[0] = 1
>>> c[None] = 2
>>> c[''] = 3
>>> list(c.elements())
[0, '', '', '', None, None]
>>> any(c.elements())
False  # FAIL

>>> any(True for i in c.elements())
True
>>> d = Counter()
>>> any(True for i in d.elements())
False

--- Bruce
*New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April
Fools!
*New!** *Blog post:
http://www.vroospeak.com/2011/04/march-gets-more-madness-next-year.html April
Fools!



On Wed, Apr 6, 2011 at 1:00 AM, cool-RR <cool-rr at cool-rr.com> wrote:

> On Wed, Apr 6, 2011 at 9:57 AM, Raymond Hettinger <
> raymond.hettinger at gmail.com> wrote:
>
>>
>> On Apr 5, 2011, at 11:59 PM, cool-RR wrote:
>>
>> > Okay, this argument applies against `Counter.__bool__`, but not against
>> `Counter.elements().__bool__`. So I want to hear whether people would find a
>>  `Counter.elements().__bool__` method useful.
>> >
>>
>>
>>
>> if any(c.elements()):
>>    ...
>>
>>
>> Raymond
>>
>
>
> Cool, this will be good enough.
>
>
> Ram.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110406/bc4ebeff/attachment.html>

From cool-rr at cool-rr.com  Wed Apr  6 16:02:45 2011
From: cool-rr at cool-rr.com (cool-RR)
Date: Wed, 6 Apr 2011 16:02:45 +0200
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <BANLkTi=vxyXnmoNuaE04z-s1hy7GR5E54Q@mail.gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
	<FF51BDBC-F20A-4884-9DAA-1C309A43CEDA@gmail.com>
	<BANLkTi=MwOUv2dXoWpoVwQt05zvamE6LOA@mail.gmail.com>
	<AA4D0E90-2E55-4627-AAED-8668E59AD9C9@gmail.com>
	<BANLkTincU+qFDvwp926qkDV+D8h3NTruXA@mail.gmail.com>
	<BANLkTi=vxyXnmoNuaE04z-s1hy7GR5E54Q@mail.gmail.com>
Message-ID: <BANLkTim7GYKZg7vOnnWYTtC1bDcfhubr_g@mail.gmail.com>

On Wed, Apr 6, 2011 at 3:57 PM, Bruce Leban <bruce at leapyear.org> wrote:

> No it won't.
>
> >>> c = Counter()
> >>> c[0] = 1
> >>> c[None] = 2
> >>> c[''] = 3
> >>> list(c.elements())
> [0, '', '', '', None, None]
> >>> any(c.elements())
> False  # FAIL
>
> >>> any(True for i in c.elements())
> True
> >>> d = Counter()
> >>> any(True for i in d.elements())
> False
>
> --- Bruce
>

Nice catch Bruce.

I think that `any(counter.values())` is solid:

>>> from collections import Counter
>>> counter = Counter([0, '', '', '', None, None])
>>> counter
Counter({'': 3, None: 2, 0: 1})
>>> any(counter.values())
True
>>> any(Counter().values())
False



Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110406/b49e1226/attachment.html>

From bruce at leapyear.org  Wed Apr  6 16:24:05 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Wed, 6 Apr 2011 07:24:05 -0700
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <BANLkTim7GYKZg7vOnnWYTtC1bDcfhubr_g@mail.gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
	<FF51BDBC-F20A-4884-9DAA-1C309A43CEDA@gmail.com>
	<BANLkTi=MwOUv2dXoWpoVwQt05zvamE6LOA@mail.gmail.com>
	<AA4D0E90-2E55-4627-AAED-8668E59AD9C9@gmail.com>
	<BANLkTincU+qFDvwp926qkDV+D8h3NTruXA@mail.gmail.com>
	<BANLkTi=vxyXnmoNuaE04z-s1hy7GR5E54Q@mail.gmail.com>
	<BANLkTim7GYKZg7vOnnWYTtC1bDcfhubr_g@mail.gmail.com>
Message-ID: <BANLkTimVVR=z0Y15YJ1XPdzRWyFpQdM5xg@mail.gmail.com>

Nope.

>>> c = Counter()
>>> c[0] = -1
>>> c.elements()
[]
>>> c.values()
dict_values([-1])
>>> any(c.values())
True  # FAIL
>>> any(True for i in c.elements())
False

--- Bruce
*New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April
Fools!
*New!** *Blog post:
http://www.vroospeak.com/2011/04/march-gets-more-madness-next-year.html April
Fools!



On Wed, Apr 6, 2011 at 7:02 AM, cool-RR <cool-rr at cool-rr.com> wrote:

> On Wed, Apr 6, 2011 at 3:57 PM, Bruce Leban <bruce at leapyear.org> wrote:
>
>> No it won't.
>>
>> >>> c = Counter()
>> >>> c[0] = 1
>> >>> c[None] = 2
>> >>> c[''] = 3
>> >>> list(c.elements())
>> [0, '', '', '', None, None]
>> >>> any(c.elements())
>> False  # FAIL
>>
>> >>> any(True for i in c.elements())
>> True
>> >>> d = Counter()
>> >>> any(True for i in d.elements())
>> False
>>
>> --- Bruce
>>
>
> Nice catch Bruce.
>
> I think that `any(counter.values())` is solid:
>
> >>> from collections import Counter
> >>> counter = Counter([0, '', '', '', None, None])
> >>> counter
> Counter({'': 3, None: 2, 0: 1})
> >>> any(counter.values())
> True
> >>> any(Counter().values())
> False
>
>
>
> Ram.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110406/0060ca21/attachment.html>

From cool-rr at cool-rr.com  Wed Apr  6 16:29:20 2011
From: cool-rr at cool-rr.com (cool-RR)
Date: Wed, 6 Apr 2011 16:29:20 +0200
Subject: [Python-ideas] A meaningful `if counter:`
In-Reply-To: <BANLkTimVVR=z0Y15YJ1XPdzRWyFpQdM5xg@mail.gmail.com>
References: <BANLkTikfeF3Kb2YdDvGrYY54ET55GBSP9A@mail.gmail.com>
	<FF51BDBC-F20A-4884-9DAA-1C309A43CEDA@gmail.com>
	<BANLkTi=MwOUv2dXoWpoVwQt05zvamE6LOA@mail.gmail.com>
	<AA4D0E90-2E55-4627-AAED-8668E59AD9C9@gmail.com>
	<BANLkTincU+qFDvwp926qkDV+D8h3NTruXA@mail.gmail.com>
	<BANLkTi=vxyXnmoNuaE04z-s1hy7GR5E54Q@mail.gmail.com>
	<BANLkTim7GYKZg7vOnnWYTtC1bDcfhubr_g@mail.gmail.com>
	<BANLkTimVVR=z0Y15YJ1XPdzRWyFpQdM5xg@mail.gmail.com>
Message-ID: <BANLkTinMbp=gce6EQAfk0MJUWeLQVuBn+g@mail.gmail.com>

On Wed, Apr 6, 2011 at 4:24 PM, Bruce Leban <bruce at leapyear.org> wrote:

> Nope.
>
> >>> c = Counter()
> >>> c[0] = -1
> >>> c.elements()
> []
> >>> c.values()
> dict_values([-1])
> >>> any(c.values())
> True  # FAIL
> >>> any(True for i in c.elements())
> False
>
> --- Bruce
>
>
Hm. Then maybe a `Counter.elements().__bool__` method would be helpful.


Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110406/7a84b460/attachment.html>

From greg.ewing at canterbury.ac.nz  Wed Apr  6 23:29:22 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 07 Apr 2011 09:29:22 +1200
Subject: [Python-ideas] Extending properties to sequence-like behavior
In-Reply-To: <BANLkTi=qK3NcL=3rBJZwHP+yYdnhyjZ0TQ@mail.gmail.com>
References: <BANLkTi=qK3NcL=3rBJZwHP+yYdnhyjZ0TQ@mail.gmail.com>
Message-ID: <4D9CDB32.1070405@canterbury.ac.nz>

Dan Baker wrote:

> The "get all" function no longer works exactly as above, though;
> box.Items would instead return the ListProp object. This can be solved
> by using __call__ on the ListProp as the "get all" method;

Alternatively you could make box.Items[:] call GetAllItems.

-- 
Greg


From dbaker3448 at gmail.com  Thu Apr  7 06:03:38 2011
From: dbaker3448 at gmail.com (Dan Baker)
Date: Wed, 6 Apr 2011 23:03:38 -0500
Subject: [Python-ideas] Extending properties to sequence-like behavior
In-Reply-To: <BANLkTi=N_06gM2M0qoGS9xxVwthyxjc3jw@mail.gmail.com>
References: <BANLkTi=qK3NcL=3rBJZwHP+yYdnhyjZ0TQ@mail.gmail.com>
	<BANLkTi=N_06gM2M0qoGS9xxVwthyxjc3jw@mail.gmail.com>
Message-ID: <BANLkTi=KmaUAEibbytWrMpendfhDE_AJqg@mail.gmail.com>

> Perhaps rename the returned class to SequenceProp and have it inherit
> from collections.MutableSequence?

I hadn't really looked around at the collections module before. If I'm
understanding it correctly, if I implement a few of the basic methods
for MutableSequence (__getitem__, __setitem__, __delitem__, and
insert) I get a few of the extra list-like methods (append, extend,
count, index, pop, etc.) and the iterator protocol for free. If that's
the case, that sounds like a huge win. Probably wouldn't even need to
do __call__ for the GetAll method anymore in most cases, I'd be able
to just use "for item in box.items" and the __iter__ method handles
it.

Thanks for the idea. That looks like it would make access via the
sequence property much more natural.

Dan


From ncoghlan at gmail.com  Thu Apr  7 06:36:33 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 7 Apr 2011 14:36:33 +1000
Subject: [Python-ideas] Extending properties to sequence-like behavior
In-Reply-To: <BANLkTi=KmaUAEibbytWrMpendfhDE_AJqg@mail.gmail.com>
References: <BANLkTi=qK3NcL=3rBJZwHP+yYdnhyjZ0TQ@mail.gmail.com>
	<BANLkTi=N_06gM2M0qoGS9xxVwthyxjc3jw@mail.gmail.com>
	<BANLkTi=KmaUAEibbytWrMpendfhDE_AJqg@mail.gmail.com>
Message-ID: <BANLkTinkkuX=DbT0Ji5BYLvzjqqOTeY02g@mail.gmail.com>

On Thu, Apr 7, 2011 at 2:03 PM, Dan Baker <dbaker3448 at gmail.com> wrote:
>> Perhaps rename the returned class to SequenceProp and have it inherit
>> from collections.MutableSequence?
>
> I hadn't really looked around at the collections module before. If I'm
> understanding it correctly, if I implement a few of the basic methods
> for MutableSequence (__getitem__, __setitem__, __delitem__, and
> insert) I get a few of the extra list-like methods (append, extend,
> count, index, pop, etc.) and the iterator protocol for free. If that's
> the case, that sounds like a huge win. Probably wouldn't even need to
> do __call__ for the GetAll method anymore in most cases, I'd be able
> to just use "for item in box.items" and the __iter__ method handles
> it.
>
> Thanks for the idea. That looks like it would make access via the
> sequence property much more natural.

Yeah, the collections ABCs were inspired in part by the old UserDict
and UserList classes - make it easier to support the broader APIs by
implementing a few essential methods. Being able to do that is one of
the big reasons Guido opted for standard library level ABC support
over Java-style interface definitions.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From jeanpierreda at gmail.com  Thu Apr  7 08:59:12 2011
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Thu, 7 Apr 2011 02:59:12 -0400
Subject: [Python-ideas] PEP-3151 pattern-matching
Message-ID: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>

Hello,

PEP-3151 < http://www.python.org/dev/peps/pep-3151/ > mentions a
really weird syntax for pattern-matching. I was hoping I could suggest
an alternative that's both more concise, and possible to implement
without doing something drastic like changing existing syntax or
semantics.

The PEP offers the pattern-matching syntax:

> except IOError as e if e.errno == errno.ENOENT: ...

I'd instead suggest something along the lines of

> except io_error(errno.ENOENT): ...

Implementing this is fairly straightforward, I've included it in the
bottom of this email. Raising an exception becomes `raise
io_error(errno.ENOENT)(msg)`. Some notational sugar can be implemented
(for example, `raise io_error(errno.ENOENT, msg)` could also be made
to work). I'm not fussed about the details.

I personally prefer keeping the errnos as a big part of handling
exceptions, they're common across OSes and involve less research /
memorization for those that are already aware of errno. I guess what
I'd like to see is the same exception hierarchy proposed by the PEP,
but with the addition of allowing errnos to be specified by
pattern-matching, so that errors not covered by the hierarchy, or more
specific than the hierarchy, can be concisely caught. However, I'm not
really well-versed in the pros and cons for all of this.

Above all, I'd like for the pattern matching alternative to be a bit
more reasonable. It doesn't have to be verbose and it doesn't have to
involve new syntax. Apologies for any mistakes in the code, they are
my own.

Here's the code:


# evil global state or something
error_classes = {}

def io_error(errno_, msg=None): # or something, you get the idea
    try:
        cls = error_classes[errno_]
    except LookupError:
        class IOErrorWithErrno(IOError):
            errno = errno_

        cls = error_classes[errno_] = IOErrorWithErrno

    return error_classes[errno_]


# example of usage
import errno
try:
    raise io_error(errno.ENOENT)("<Error message>")
except io_error(errno.ENOENT):
    print("success")


Thanks for your time!
Devin Jeanpierre


From mal at egenix.com  Thu Apr  7 10:43:25 2011
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 07 Apr 2011 10:43:25 +0200
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
Message-ID: <4D9D792D.2020403@egenix.com>

Devin Jeanpierre wrote:
> Hello,
> 
> PEP-3151 < http://www.python.org/dev/peps/pep-3151/ > mentions a
> really weird syntax for pattern-matching. I was hoping I could suggest
> an alternative that's both more concise, and possible to implement
> without doing something drastic like changing existing syntax or
> semantics.
> 
> The PEP offers the pattern-matching syntax:
> 
>> except IOError as e if e.errno == errno.ENOENT: ...
> 
> I'd instead suggest something along the lines of
> 
>> except io_error(errno.ENOENT): ...
> 
> Implementing this is fairly straightforward, I've included it in the
> bottom of this email. Raising an exception becomes `raise
> io_error(errno.ENOENT)(msg)`. Some notational sugar can be implemented
> (for example, `raise io_error(errno.ENOENT, msg)` could also be made
> to work). I'm not fussed about the details.

The major difference between your proposal and the one in the
PEP is that in your case, an error object has to be created
or looked up via a function call regardless of whether the
case matches or not.

You typically don't want that to happen in tight loops and
except-clauses are designed to be cheap if they don't match.

> I personally prefer keeping the errnos as a big part of handling
> exceptions, they're common across OSes and involve less research /
> memorization for those that are already aware of errno. I guess what
> I'd like to see is the same exception hierarchy proposed by the PEP,
> but with the addition of allowing errnos to be specified by
> pattern-matching, so that errors not covered by the hierarchy, or more
> specific than the hierarchy, can be concisely caught. However, I'm not
> really well-versed in the pros and cons for all of this.

The main problem with the PEP is the proposal to flatten existing
exception class hierarchies, e.g. making socket.error the same
as IOError.

This introduces very subtle compatibility problems with code that
uses the flattened exception classes in separate except branches, e.g.

try:
 ...
except socket.error:
 ...
except IOError:
 ...

With the PEP implemented, the above code would never get
to execute the IOError branch and instead divert all error
handling to the socket.error branch which may well not be
aware of all the extra cases it now has to handle.

I think that this part of the PEP should not be accepted.

The addition of new IOError subclasses for common errno cases
is useful.

It would be even more useful, if there were a way to catch
standard IOErrors with those errnos using those same classes,
so that the following becomes possible:

try:
 ...
 raise IOError(errno.EPERM, "permission denied")
except PermissionError:
 ...

and works as one would expect, that is, catch the EPERM error.

Without such support, you'd still have to write:

try:
 ...
 raise IOError(errno.EPERM, "permission denied")
except PermissionError:
 ...EPERM handling code...
except IOError as error:
 if error.errno == errno.EPERM:
   ...EPERM handling code...

duplicating the error handling code.

Perhaps the IOError constructor could be made to switch
the class of the generated object based on the errno
attribute passed to the constructor.

That way no new syntax would be necessary at all.

> Above all, I'd like for the pattern matching alternative to be a bit
> more reasonable. It doesn't have to be verbose and it doesn't have to
> involve new syntax. Apologies for any mistakes in the code, they are
> my own.
> 
> Here's the code:
> 
> 
> # evil global state or something
> error_classes = {}
> 
> def io_error(errno_, msg=None): # or something, you get the idea
>     try:
>         cls = error_classes[errno_]
>     except LookupError:
>         class IOErrorWithErrno(IOError):
>             errno = errno_
> 
>         cls = error_classes[errno_] = IOErrorWithErrno
> 
>     return error_classes[errno_]
> 
> 
> # example of usage
> import errno
> try:
>     raise io_error(errno.ENOENT)("<Error message>")
> except io_error(errno.ENOENT):
>     print("success")
> 
> 
> Thanks for your time!
> Devin Jeanpierre

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 07 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/


From solipsis at pitrou.net  Thu Apr  7 11:17:16 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 7 Apr 2011 11:17:16 +0200
Subject: [Python-ideas] PEP-3151 pattern-matching
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<4D9D792D.2020403@egenix.com>
Message-ID: <20110407111716.47f49883@pitrou.net>

On Thu, 07 Apr 2011 10:43:25 +0200
"M.-A. Lemburg" <mal at egenix.com> wrote:
> 
> It would be even more useful, if there were a way to catch
> standard IOErrors with those errnos using those same classes,
> so that the following becomes possible:
> 
> try:
>  ...
>  raise IOError(errno.EPERM, "permission denied")
> except PermissionError:
>  ...
> 
> and works as one would expect, that is, catch the EPERM error.

> Perhaps the IOError constructor could be made to switch
> the class of the generated object based on the errno
> attribute passed to the constructor.

Nice suggestion. I'll try to see if that is possible.

Regards

Antoine.




From ncoghlan at gmail.com  Thu Apr  7 13:48:01 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 7 Apr 2011 21:48:01 +1000
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <20110407111716.47f49883@pitrou.net>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<4D9D792D.2020403@egenix.com> <20110407111716.47f49883@pitrou.net>
Message-ID: <BANLkTini1t+KEvMdtbq7SNujLXuGe7OmPA@mail.gmail.com>

On Thu, Apr 7, 2011 at 7:17 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> Perhaps the IOError constructor could be made to switch
>> the class of the generated object based on the errno
>> attribute passed to the constructor.
>
> Nice suggestion. I'll try to see if that is possible.

It should be possible with appropriate fancy footwork in __new__. You
do need to be careful to avoid calling __init__ on the created object
twice.

I know I've read an example that demonstrates the principle, but I
unfortunately don't remember where (I initially thought it was in
Guido's new-style class essay, but I checked and that wasn't it)

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From fuzzyman at gmail.com  Thu Apr  7 15:32:38 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Thu, 7 Apr 2011 14:32:38 +0100
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTini1t+KEvMdtbq7SNujLXuGe7OmPA@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<4D9D792D.2020403@egenix.com> <20110407111716.47f49883@pitrou.net>
	<BANLkTini1t+KEvMdtbq7SNujLXuGe7OmPA@mail.gmail.com>
Message-ID: <BANLkTimnw=-mj+mMFQr3Abz6svfj57PwhA@mail.gmail.com>

On 7 April 2011 12:48, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Thu, Apr 7, 2011 at 7:17 PM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> >> Perhaps the IOError constructor could be made to switch
> >> the class of the generated object based on the errno
> >> attribute passed to the constructor.
> >
> > Nice suggestion. I'll try to see if that is possible.
>
> It should be possible with appropriate fancy footwork in __new__. You
> do need to be careful to avoid calling __init__ on the created object
> twice.
>
> I know I've read an example that demonstrates the principle, but I
> unfortunately don't remember where (I initially thought it was in
> Guido's new-style class essay, but I checked and that wasn't it)
>


__init__ is called for you on construction so long as the object returned by
__new__ is an instance of the type being constructed or a subclass.

So if what you want __new__ to return *is* a subclass, then you create it
with subclass.__new__(...) and not subclass(...) (the latter would call
__init__ which would then be called again after __new__ returns).

If what you're returning *isn't* a subclass (which is best avoided anyway)
then you can construct it with otherclass(...) as __init__ won't be called
for you.

All the best,

Michael Foord


>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110407/f92219d3/attachment.html>

From fuzzyman at gmail.com  Thu Apr  7 15:33:16 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Thu, 7 Apr 2011 14:33:16 +0100
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTimnw=-mj+mMFQr3Abz6svfj57PwhA@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<4D9D792D.2020403@egenix.com> <20110407111716.47f49883@pitrou.net>
	<BANLkTini1t+KEvMdtbq7SNujLXuGe7OmPA@mail.gmail.com>
	<BANLkTimnw=-mj+mMFQr3Abz6svfj57PwhA@mail.gmail.com>
Message-ID: <BANLkTimdENE-ONFp2v7diwGXoAtXqrwHwA@mail.gmail.com>

On 7 April 2011 14:32, Michael Foord <fuzzyman at gmail.com> wrote:

>
>
> On 7 April 2011 12:48, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On Thu, Apr 7, 2011 at 7:17 PM, Antoine Pitrou <solipsis at pitrou.net>
>> wrote:
>> >> Perhaps the IOError constructor could be made to switch
>> >> the class of the generated object based on the errno
>> >> attribute passed to the constructor.
>> >
>> > Nice suggestion. I'll try to see if that is possible.
>>
>> It should be possible with appropriate fancy footwork in __new__. You
>> do need to be careful to avoid calling __init__ on the created object
>> twice.
>>
>> I know I've read an example that demonstrates the principle, but I
>> unfortunately don't remember where (I initially thought it was in
>> Guido's new-style class essay, but I checked and that wasn't it)
>>
>
>
> __init__ is called for you on construction so long as the object returned
> by __new__ is an instance of the type being constructed or a subclass.
>
> So if what you want __new__ to return *is* a subclass, then you create it
> with subclass.__new__(...) and not subclass(...) (the latter would call
> __init__ which would then be called again after __new__ returns).
>
> If what you're returning *isn't* a subclass (which is best avoided anyway)
> then you can construct it with otherclass(...) as __init__ won't be called
> for you.
>
>
Those are the pure python rules anyway (which you were probably aware of),
no idea if it is different in C. :-)

Michael


> All the best,
>
> Michael Foord
>
>
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
>
>
> --
>
> http://www.voidspace.org.uk/
>
> May you do good and not evil
> May you find forgiveness for yourself and forgive others
>
> May you share freely, never taking more than you give.
> -- the sqlite blessing http://www.sqlite.org/different.html
>
>
>


-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110407/f2489380/attachment.html>

From fuzzyman at gmail.com  Thu Apr  7 15:48:42 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Thu, 7 Apr 2011 14:48:42 +0100
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTimnw=-mj+mMFQr3Abz6svfj57PwhA@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<4D9D792D.2020403@egenix.com> <20110407111716.47f49883@pitrou.net>
	<BANLkTini1t+KEvMdtbq7SNujLXuGe7OmPA@mail.gmail.com>
	<BANLkTimnw=-mj+mMFQr3Abz6svfj57PwhA@mail.gmail.com>
Message-ID: <BANLkTi=MruAAs2pE-i-MjW3pXNyANQA9=g@mail.gmail.com>

On 7 April 2011 14:32, Michael Foord <fuzzyman at gmail.com> wrote:

>
>
> On 7 April 2011 12:48, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On Thu, Apr 7, 2011 at 7:17 PM, Antoine Pitrou <solipsis at pitrou.net>
>> wrote:
>> >> Perhaps the IOError constructor could be made to switch
>> >> the class of the generated object based on the errno
>> >> attribute passed to the constructor.
>> >
>> > Nice suggestion. I'll try to see if that is possible.
>>
>> It should be possible with appropriate fancy footwork in __new__. You
>> do need to be careful to avoid calling __init__ on the created object
>> twice.
>>
>> I know I've read an example that demonstrates the principle, but I
>> unfortunately don't remember where (I initially thought it was in
>> Guido's new-style class essay, but I checked and that wasn't it)
>>
>
>
> __init__ is called for you on construction so long as the object returned
> by __new__ is an instance of the type being constructed or a subclass.
>
> So if what you want __new__ to return *is* a subclass, then you create it
> with subclass.__new__(...)
>

Hmmm... that would rely on subclass.__new__ both existing *and* not calling
up to its parent __new__ or you will have infinite recursion.

Probably what you have to do is call object.__new__(subclass, ...) and
knowing / praying that subclass.__new__ doesn't do anything important...

All the best,

Michael Foord


> and not subclass(...) (the latter would call __init__ which would then be
> called again after __new__ returns).
>
> If what you're returning *isn't* a subclass (which is best avoided anyway)
> then you can construct it with otherclass(...) as __init__ won't be called
> for you.
>
> All the best,
>
> Michael Foord
>
>
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
>
>
> --
>
> http://www.voidspace.org.uk/
>
> May you do good and not evil
> May you find forgiveness for yourself and forgive others
>
> May you share freely, never taking more than you give.
> -- the sqlite blessing http://www.sqlite.org/different.html
>
>
>


-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110407/f2db325f/attachment.html>

From ncoghlan at gmail.com  Thu Apr  7 16:36:39 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 8 Apr 2011 00:36:39 +1000
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTi=MruAAs2pE-i-MjW3pXNyANQA9=g@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<4D9D792D.2020403@egenix.com> <20110407111716.47f49883@pitrou.net>
	<BANLkTini1t+KEvMdtbq7SNujLXuGe7OmPA@mail.gmail.com>
	<BANLkTimnw=-mj+mMFQr3Abz6svfj57PwhA@mail.gmail.com>
	<BANLkTi=MruAAs2pE-i-MjW3pXNyANQA9=g@mail.gmail.com>
Message-ID: <BANLkTin=YNPe0x4xqbansXDx7ptCDDoObg@mail.gmail.com>

On Thu, Apr 7, 2011 at 11:48 PM, Michael Foord <fuzzyman at gmail.com> wrote:
> Hmmm... that would rely on subclass.__new__ both existing *and* not calling
> up to its parent __new__ or you will have infinite recursion.
> Probably what you have to do is call object.__new__(subclass, ...) and
> knowing / praying that subclass.__new__ doesn't do anything important...

That's the dance I'm trying to remember. You make it work by playing
identity checking games with the cls argument, but it's been
absolutely ages since I read about it and experimented with it.

I think it's something like:

def __new__(cls, *args, **kwds):
  if cls is ThisClass:
    # Do fancy footwork to implicitly create an appropriate subclass instead
    # via subclass.__new__
    obj = cls._create_instance(*args, **kwds)
  else:
    # Don't do anything tricky for subclasses, that's their problem
    obj = object.__new__(*args, **kwds)
  return obj

Subclasses then have the option of passing the parent class as "cls"
if they want to invoke the fancy footwork, or themselves if they
don't.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From fuzzyman at gmail.com  Thu Apr  7 17:53:23 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Thu, 7 Apr 2011 16:53:23 +0100
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTin=YNPe0x4xqbansXDx7ptCDDoObg@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<4D9D792D.2020403@egenix.com> <20110407111716.47f49883@pitrou.net>
	<BANLkTini1t+KEvMdtbq7SNujLXuGe7OmPA@mail.gmail.com>
	<BANLkTimnw=-mj+mMFQr3Abz6svfj57PwhA@mail.gmail.com>
	<BANLkTi=MruAAs2pE-i-MjW3pXNyANQA9=g@mail.gmail.com>
	<BANLkTin=YNPe0x4xqbansXDx7ptCDDoObg@mail.gmail.com>
Message-ID: <BANLkTi=mEs+9yNccDt21VVaXZg2zF+8ngQ@mail.gmail.com>

On 7 April 2011 15:36, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Thu, Apr 7, 2011 at 11:48 PM, Michael Foord <fuzzyman at gmail.com> wrote:
> > Hmmm... that would rely on subclass.__new__ both existing *and* not
> calling
> > up to its parent __new__ or you will have infinite recursion.
> > Probably what you have to do is call object.__new__(subclass, ...) and
> > knowing / praying that subclass.__new__ doesn't do anything important...
>
> That's the dance I'm trying to remember. You make it work by playing
> identity checking games with the cls argument, but it's been
> absolutely ages since I read about it and experimented with it.
>
> I think it's something like:
>
> def __new__(cls, *args, **kwds):
>  if cls is ThisClass:
>    # Do fancy footwork to implicitly create an appropriate subclass instead
>    # via subclass.__new__
>    obj = cls._create_instance(*args, **kwds)
>  else:
>    # Don't do anything tricky for subclasses, that's their problem
>    obj = object.__new__(*args, **kwds)
>  return obj
>
> Subclasses then have the option of passing the parent class as "cls"
> if they want to invoke the fancy footwork, or themselves if they
> don't.
>
>
Nice solution. You should write it up on your blog. It lets you call
subclass.__new__, to return instances of subclasses, without having to worry
about whether or not subclass.__new__ is going to upcall.

Michael


> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110407/6fc797f7/attachment.html>

From jeanpierreda at gmail.com  Thu Apr  7 19:44:10 2011
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Thu, 7 Apr 2011 13:44:10 -0400
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTi=mEs+9yNccDt21VVaXZg2zF+8ngQ@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<4D9D792D.2020403@egenix.com> <20110407111716.47f49883@pitrou.net>
	<BANLkTini1t+KEvMdtbq7SNujLXuGe7OmPA@mail.gmail.com>
	<BANLkTimnw=-mj+mMFQr3Abz6svfj57PwhA@mail.gmail.com>
	<BANLkTi=MruAAs2pE-i-MjW3pXNyANQA9=g@mail.gmail.com>
	<BANLkTin=YNPe0x4xqbansXDx7ptCDDoObg@mail.gmail.com>
	<BANLkTi=mEs+9yNccDt21VVaXZg2zF+8ngQ@mail.gmail.com>
Message-ID: <BANLkTimpJdAq3_bafaxj7du5doTUaxCovg@mail.gmail.com>

On Thu, Apr 7, 2011 at 4:43 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> You typically don't want that to happen in tight loops and
> except-clauses are designed to be cheap if they don't match.

Ah, well, you can always except IOError directly if performance is important.

Also, sorry, I left an error in my code. `msg=None` should be omitted.

Nonetheless it appears my idea has been discarded by now. Doesn't
matter too much, just wanted to get it off my chest. Thanks for the
time, everyone!

Devin Jeanpierre


From ncoghlan at gmail.com  Fri Apr  8 01:28:31 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 8 Apr 2011 09:28:31 +1000
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <4D9D792D.2020403@egenix.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<4D9D792D.2020403@egenix.com>
Message-ID: <BANLkTinJ6pHxeY+sUbce+Qfqyc+MNTVO9A@mail.gmail.com>

On Thu, Apr 7, 2011 at 6:43 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> The main problem with the PEP is the proposal to flatten existing
> exception class hierarchies, e.g. making socket.error the same
> as IOError.
>
> This introduces very subtle compatibility problems with code that
> uses the flattened exception classes in separate except branches, e.g.
>
> try:
> ?...
> except socket.error:
> ?...
> except IOError:
> ?...
>
> With the PEP implemented, the above code would never get
> to execute the IOError branch and instead divert all error
> handling to the socket.error branch which may well not be
> aware of all the extra cases it now has to handle.
>
> I think that this part of the PEP should not be accepted.

I think EnvironmentError, WindowsError, VMSError, OSError, mmap.error
and select.error should definitely all be merged with IOError, as they
aren't used consistently enough to make handling them differently
reliable even in current code.

socket.error is the one case where that argument isn't quite as
strong, since the socket module is reasonably consistent about raising
it over a base IOError.

I think it would be useful to look at which errno values may currently
be associated with socket.error, and merge it with the new exception
at that level in the type heirarchy.

For example, taking the draft heirarchy at
http://www.python.org/dev/peps/pep-3151/#new-exception-classes, then
the merger that makes the most sense might be "socket.error =
ConnectionError".

Alternatively, and if necessary, socket.error could be grandfathered
in to cover an appropriate subset of errno values via multiple
inheritance: class error(ConnectionError, TimeoutError): pass

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From mal at egenix.com  Fri Apr  8 10:59:16 2011
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 08 Apr 2011 10:59:16 +0200
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTinJ6pHxeY+sUbce+Qfqyc+MNTVO9A@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>	<4D9D792D.2020403@egenix.com>
	<BANLkTinJ6pHxeY+sUbce+Qfqyc+MNTVO9A@mail.gmail.com>
Message-ID: <4D9ECE64.7070402@egenix.com>

Nick Coghlan wrote:
> On Thu, Apr 7, 2011 at 6:43 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> The main problem with the PEP is the proposal to flatten existing
>> exception class hierarchies, e.g. making socket.error the same
>> as IOError.
>>
>> This introduces very subtle compatibility problems with code that
>> uses the flattened exception classes in separate except branches, e.g.
>>
>> try:
>>  ...
>> except socket.error:
>>  ...
>> except IOError:
>>  ...
>>
>> With the PEP implemented, the above code would never get
>> to execute the IOError branch and instead divert all error
>> handling to the socket.error branch which may well not be
>> aware of all the extra cases it now has to handle.
>>
>> I think that this part of the PEP should not be accepted.
> 
> I think EnvironmentError, WindowsError, VMSError, OSError, mmap.error
> and select.error should definitely all be merged with IOError, as they
> aren't used consistently enough to make handling them differently
> reliable even in current code.

Their use may be inconsistent in a few places, but those cases
are still well-defined by the implementation, so code relying
on that well-defined behavior will break in subtle ways.

Example:

If code catches select.error (which is only raised for select.select()
and poll_object.poll() calls), because the try...except is expecting
a possible error from the select call used in the try...except
block, it will most likely do the wrong thing for some IOError
raised by a logging file I/O call in that same block.

Moreover, catching the exception at the wrong level, will
prevent the IOError from bubbling up the call chain and can
effectively mask potential errors at lower levels.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 08 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/


From ncoghlan at gmail.com  Fri Apr  8 11:27:22 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 8 Apr 2011 19:27:22 +1000
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <4D9ECE64.7070402@egenix.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<4D9D792D.2020403@egenix.com>
	<BANLkTinJ6pHxeY+sUbce+Qfqyc+MNTVO9A@mail.gmail.com>
	<4D9ECE64.7070402@egenix.com>
Message-ID: <BANLkTikWnvgr95XGRe1uXzRUigTTk3SMWg@mail.gmail.com>

On Fri, Apr 8, 2011 at 6:59 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> Their use may be inconsistent in a few places, but those cases
> are still well-defined by the implementation, so code relying
> on that well-defined behavior will break in subtle ways.

The phrases "defined by the implementation" and "well-defined" do not
belong in the same sentence.

> Example:
>
> If code catches select.error (which is only raised for select.select()
> and poll_object.poll() calls), because the try...except is expecting
> a possible error from the select call used in the try...except
> block, it will most likely do the wrong thing for some IOError
> raised by a logging file I/O call in that same block.

A deprecation period for the merged exceptions would be advisable.
That's an argument in favouring of figuring out how to implement
correct deprecation warnings for the merges, not an argument in favour
of not doing them.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From mal at egenix.com  Fri Apr  8 11:53:42 2011
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 08 Apr 2011 11:53:42 +0200
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTikWnvgr95XGRe1uXzRUigTTk3SMWg@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>	<4D9D792D.2020403@egenix.com>	<BANLkTinJ6pHxeY+sUbce+Qfqyc+MNTVO9A@mail.gmail.com>	<4D9ECE64.7070402@egenix.com>
	<BANLkTikWnvgr95XGRe1uXzRUigTTk3SMWg@mail.gmail.com>
Message-ID: <4D9EDB26.3030701@egenix.com>

Nick Coghlan wrote:
> On Fri, Apr 8, 2011 at 6:59 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> Their use may be inconsistent in a few places, but those cases
>> are still well-defined by the implementation, so code relying
>> on that well-defined behavior will break in subtle ways.
> 
> The phrases "defined by the implementation" and "well-defined" do not
> belong in the same sentence.

I agree that exceptions being raised by certain APIs
should probably be documented, but in the absence of such documentation,
the implementation is still the ultimate source of wisdom and
given that it's written down in C and not some legal code, I
think "well-defined" is an appropriate term ;-)

>> Example:
>>
>> If code catches select.error (which is only raised for select.select()
>> and poll_object.poll() calls), because the try...except is expecting
>> a possible error from the select call used in the try...except
>> block, it will most likely do the wrong thing for some IOError
>> raised by a logging file I/O call in that same block.
> 
> A deprecation period for the merged exceptions would be advisable.
> That's an argument in favouring of figuring out how to implement
> correct deprecation warnings for the merges, not an argument in favour
> of not doing them.

Indeed, getting deprecations right is yet another aspect to consider,
but not the one the example was supposed to explain.

I don't think that such major changes in class hierarchy can be
implemented in a minor Python release.

Note that select.error currently does not inherit from IOError,
so "except IOError" won't catch select errors.

Renaming the exceptions would be something we could do for a
minor release and then have deprecations hint the user to
the naming change.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 08 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/


From ncoghlan at gmail.com  Fri Apr  8 13:32:47 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 8 Apr 2011 21:32:47 +1000
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
Message-ID: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>

(Oops, let's try that again with the correct destination address this time...)

A few odds and ends from recent discussions finally clicked into
something potentially interesting earlier this evening. Or possibly
just something insane. I'm not quite decided on that point as yet (but
leaning towards the latter).

Anyway, without further ado, I present:

AST Transformation Hooks for Domain Specific Languages
======================================================

Consider:

# In some other module
ast.register_dsl("dsl.sql", dsl.sql.TransformAST)

# In a module using that DSL
import dsl.sql
def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql:
   select address
   from people
   where name = {name} and dob = {dob}


Suppose that the standard AST for the latter looked something like:

   DSL(syntax="dsl.sql",
       name='lookup_address',
       args=arguments(
           args=[arg(arg='name',
                     annotation=<Normal AST for "dsl.sql.char">),
                 arg(arg='dob',
                     annotation=<Normal AST for "dsl.sql.date">)],
           vararg=None, varargannotation=None,
           kwonlyargs=[], kwarg=None, kwargannotation=None,
           defaults=[], kw_defaults=[]),
       body=[Expr(value=Str(s='select address\nfrom people\nwhere
name = {name} and dob = {dob}'))],
       decorator_list=[],
       returns=None)

(For those not familiar with the AST, the above is actually just the
existing Function node with a "syntax" attribute added)

At *compile* time (note, *not* function definition time), the
registered AST transformation hook would be invoked and would replace
that DSL node with "standard" AST nodes.

For example, depending on the design of the DSL and its support code,
the above example might be equivalent to:

   @dsl.sql.escape_and_validate_args
   def lookup_address(name: dsl.sql.char, dob: dsl.sql.date):
      args = dict(name=name, dob=dob)
      query = "select address\nfrom people\nwhere name = {name} and
dob = {dob}"
      return dsl.sql.cursor(query, args)


As a simpler example, consider something like:

   def f() from all_nonlocal:
       x += 1
       y -= 2

That was translated at compile time into:

   def f():
       nonlocal x, y
       x += 1
       y -= 2

My first pass at a rough protocol for the AST transformers suggests
they would only need two methods:

 get_cookie() - Magic cookie to add to PYC files containing instances
of the DSL (allows recompilation to be forced if the DSL is updated)
 transform_AST(node) - a DSL() node is passed in, expected to return
an AST containing no DSL nodes (SyntaxError if one is found)

Attempts to use an unregistered DSL would trigger SyntaxError

So there you are, that's the crazy idea. The stoning of the heretic
may now commence :)

Where this idea came from was the various discussions about "make
statement" style constructs and a conversation I had with Eric Snow at
Pycon about function definition time really being *too late* to do
anything particularly interesting that couldn't already be handled
better in other ways. Some tricks Dave Malcolm had done to support
Python level manipulation of the AST during compilation also played a
big part, as did Eugene Toder's efforts to add an AST optimisation
step to the compilation process.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Fri Apr  8 14:58:29 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 8 Apr 2011 22:58:29 +1000
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
Message-ID: <BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>

On Fri, Apr 8, 2011 at 9:32 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> ?get_cookie() - Magic cookie to add to PYC files containing instances
> of the DSL (allows recompilation to be forced if the DSL is updated)

An alternative might be to require that the cookie be provided when
the DSL is registered. That would make cookie validity checking
faster. That kind of implementation detail is in the noise though,
compared to the possible implications of the overall idea.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From fuzzyman at gmail.com  Fri Apr  8 15:09:37 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Fri, 8 Apr 2011 14:09:37 +0100
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
Message-ID: <BANLkTi=5-dqv_fphfGi+bNUxL5RC=wjn3Q@mail.gmail.com>

On 8 April 2011 13:58, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Fri, Apr 8, 2011 at 9:32 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >  get_cookie() - Magic cookie to add to PYC files containing instances
> > of the DSL (allows recompilation to be forced if the DSL is updated)
>
> An alternative might be to require that the cookie be provided when
> the DSL is registered. That would make cookie validity checking
> faster. That kind of implementation detail is in the noise though,
> compared to the possible implications of the overall idea.
>
>
You *really* ought to implement this as an extension module and an import
hook so that we can try it out. :-)

All the best,

Michael Foord


> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110408/8122a3e4/attachment.html>

From eltoder at gmail.com  Fri Apr  8 15:15:44 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Fri, 8 Apr 2011 09:15:44 -0400
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
Message-ID: <BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>

One of the tricky details is where to put ast.register() so that it
runs before module is parsed. Doing this in dsl.sql and importing it
in 'sql' modules is not enough. Sql modules will have to rely on
someone registering dsl early enough, e.g. with

import dsl.sql.register # register dsl
import sqlmodule # now import module using sql dsl

So registering dsl will be client's responsibility, rather than
something module can do for itself.
If this is OK, we can achieve similar effect without any changes to
Python -- for example, with import hooks. One can write a hook that
applies whatever AST transformations to modules loaded from specific
locations.

We can make AST transformation a part of module itself, e.g. with some
kind of "eager decorator". Taking your example:

@@dsl.sql.query
def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql:
   select address
   from people
   where name = {name} and dob = {dob}

Eager decorator has to be used by the fully qualified name. Parser
will import (and execute) defining module (dsl.sql in this example)
while compiling a module that uses it (not when module is executed, as
with normal decorator).

Eugene


From ncoghlan at gmail.com  Fri Apr  8 15:44:28 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 8 Apr 2011 23:44:28 +1000
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>
Message-ID: <BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>

On Fri, Apr 8, 2011 at 11:15 PM, Eugene Toder <eltoder at gmail.com> wrote:
> So registering dsl will be client's responsibility, rather than
> something module can do for itself.

Yep, but if you're at the point of using a DSL, that's likely to be
part of a larger framework which can take care of registering the DSL
for you before you import any modules that need it.

That answer applies whether this is a standard language feature or
part of an import hook that uses its own custom compiler.

> If this is OK, we can achieve similar effect without any changes to
> Python -- for example, with import hooks. One can write a hook that
> applies whatever AST transformations to modules loaded from specific
> locations.

Yeah, the big downside is having to almost completely reimplement the
AST compiler in order to do it that way (since the built-in one would
choke on the syntax extension). That isn't *hard* so much as it is
tedious (especially compared to tweaking the existing one in place on
a clone of the main repo).

Once the AST has been transformed, of course, the existing compiler
could still be used.

Note something I haven't looked into yet is whether or not the CPython
parser can even generate a different node type for this, or manage the
"automatic stringification" of the DSL body. The former issue wouldn't
be too hard to handle (just add a "syntax" attribute to the existing
Function node, with "node.syntax is None" indicating standard Python
code, but I'm not sure about the second one (although worst case would
be to require use of the docstring to cover anything that didn't fit
with standard Python syntax - arguably not a bad idea anyway, since it
would be a lot friendlier to non-DSL aware Python tools).

> We can make AST transformation a part of module itself, e.g. with some
> kind of "eager decorator". Taking your example:
>
> @@dsl.sql.query
> def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql:
> ? select address
> ? from people
> ? where name = {name} and dob = {dob}
>
> Eager decorator has to be used by the fully qualified name. Parser
> will import (and execute) defining module (dsl.sql in this example)
> while compiling a module that uses it (not when module is executed, as
> with normal decorator).

But what would the eager decorator buy you over just specifying a
different dialect in the "from" clause?

I'm not sure I'll ever actually create a prototype of this (lots of
other things on the to-do list), but I found the idea too intriguing
not to share it.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From fuzzyman at gmail.com  Fri Apr  8 16:01:21 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Fri, 8 Apr 2011 15:01:21 +0100
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
Message-ID: <BANLkTinemYibPRNMEeQMOB=YEyqjdVR0Rw@mail.gmail.com>

On 8 April 2011 12:32, Nick Coghlan <ncoghlan at gmail.com> wrote:

> (Oops, let's try that again with the correct destination address this
> time...)
>
> A few odds and ends from recent discussions finally clicked into
> something potentially interesting earlier this evening. Or possibly
> just something insane. I'm not quite decided on that point as yet (but
> leaning towards the latter).
>
> Anyway, without further ado, I present:
>

Oops, sent my reply to the wrong list as well.

The essence of the proposal is to allow arbitrary syntax within "standard
python files". I don't think it stands much of a chance in core.

It would be an awesome tool for experimenting with new syntax and DSLs
though. :-)

Michael




>
> AST Transformation Hooks for Domain Specific Languages
> ======================================================
>
> Consider:
>
> # In some other module
> ast.register_dsl("dsl.sql", dsl.sql.TransformAST)
>
> # In a module using that DSL
> import dsl.sql
> def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql:
>   select address
>   from people
>   where name = {name} and dob = {dob}
>
>
> Suppose that the standard AST for the latter looked something like:
>
>   DSL(syntax="dsl.sql",
>       name='lookup_address',
>       args=arguments(
>           args=[arg(arg='name',
>                     annotation=<Normal AST for "dsl.sql.char">),
>                 arg(arg='dob',
>                     annotation=<Normal AST for "dsl.sql.date">)],
>           vararg=None, varargannotation=None,
>           kwonlyargs=[], kwarg=None, kwargannotation=None,
>           defaults=[], kw_defaults=[]),
>       body=[Expr(value=Str(s='select address\nfrom people\nwhere
> name = {name} and dob = {dob}'))],
>       decorator_list=[],
>       returns=None)
>
> (For those not familiar with the AST, the above is actually just the
> existing Function node with a "syntax" attribute added)
>
> At *compile* time (note, *not* function definition time), the
> registered AST transformation hook would be invoked and would replace
> that DSL node with "standard" AST nodes.
>
> For example, depending on the design of the DSL and its support code,
> the above example might be equivalent to:
>
>   @dsl.sql.escape_and_validate_args
>   def lookup_address(name: dsl.sql.char, dob: dsl.sql.date):
>      args = dict(name=name, dob=dob)
>      query = "select address\nfrom people\nwhere name = {name} and
> dob = {dob}"
>      return dsl.sql.cursor(query, args)
>
>
> As a simpler example, consider something like:
>
>   def f() from all_nonlocal:
>       x += 1
>       y -= 2
>
> That was translated at compile time into:
>
>   def f():
>       nonlocal x, y
>       x += 1
>       y -= 2
>
> My first pass at a rough protocol for the AST transformers suggests
> they would only need two methods:
>
>  get_cookie() - Magic cookie to add to PYC files containing instances
> of the DSL (allows recompilation to be forced if the DSL is updated)
>  transform_AST(node) - a DSL() node is passed in, expected to return
> an AST containing no DSL nodes (SyntaxError if one is found)
>
> Attempts to use an unregistered DSL would trigger SyntaxError
>
> So there you are, that's the crazy idea. The stoning of the heretic
> may now commence :)
>
> Where this idea came from was the various discussions about "make
> statement" style constructs and a conversation I had with Eric Snow at
> Pycon about function definition time really being *too late* to do
> anything particularly interesting that couldn't already be handled
> better in other ways. Some tricks Dave Malcolm had done to support
> Python level manipulation of the AST during compilation also played a
> big part, as did Eugene Toder's efforts to add an AST optimisation
> step to the compilation process.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110408/7d30cbcb/attachment.html>

From mikegraham at gmail.com  Fri Apr  8 16:04:50 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Fri, 8 Apr 2011 10:04:50 -0400
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
Message-ID: <BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>

On Thu, Apr 7, 2011 at 2:59 AM, Devin Jeanpierre <jeanpierreda at gmail.com> wrote:
> Hello,
>
> PEP-3151 < http://www.python.org/dev/peps/pep-3151/ > mentions a
> really weird syntax for pattern-matching. I was hoping I could suggest
> an alternative that's both more concise, and possible to implement
> without doing something drastic like changing existing syntax or
> semantics.
>
> The PEP offers the pattern-matching syntax:
>
>> except IOError as e if e.errno == errno.ENOENT: ...
>
> I'd instead suggest something along the lines of
>
>> except io_error(errno.ENOENT): ...
>
> Implementing this is fairly straightforward, I've included it in the
> bottom of this email. Raising an exception becomes `raise
> io_error(errno.ENOENT)(msg)`. Some notational sugar can be implemented
> (for example, `raise io_error(errno.ENOENT, msg)` could also be made
> to work). I'm not fussed about the details.
>
> I personally prefer keeping the errnos as a big part of handling
> exceptions, they're common across OSes and involve less research /
> memorization for those that are already aware of errno. I guess what
> I'd like to see is the same exception hierarchy proposed by the PEP,
> but with the addition of allowing errnos to be specified by
> pattern-matching, so that errors not covered by the hierarchy, or more
> specific than the hierarchy, can be concisely caught. However, I'm not
> really well-versed in the pros and cons for all of this.
>
> Above all, I'd like for the pattern matching alternative to be a bit
> more reasonable. It doesn't have to be verbose and it doesn't have to
> involve new syntax. Apologies for any mistakes in the code, they are
> my own.
>
> Here's the code:
>
>
> # evil global state or something
> error_classes = {}
>
> def io_error(errno_, msg=None): # or something, you get the idea
> ? ?try:
> ? ? ? ?cls = error_classes[errno_]
> ? ?except LookupError:
> ? ? ? ?class IOErrorWithErrno(IOError):
> ? ? ? ? ? ?errno = errno_
>
> ? ? ? ?cls = error_classes[errno_] = IOErrorWithErrno
>
> ? ?return error_classes[errno_]
>
>
> # example of usage
> import errno
> try:
> ? ?raise io_error(errno.ENOENT)("<Error message>")
> except io_error(errno.ENOENT):
> ? ?print("success")
>
>
> Thanks for your time!
> Devin Jeanpierre

I don't see how either solution is better than continuing to do what
we have right now. The PEP's idea introduces new syntax for a problem
that is currently solved in two lines. Your suggestion makes a new
pattern within the Python exception world, and further helps make the
exception hierarchy a little harder to understand again.

Neither of these seem justified for a rare case (this sort of patter
is fairly rare, most notably this one example) when there's nothing
that awful about the current solution.

Mike


From ncoghlan at gmail.com  Fri Apr  8 16:13:41 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 9 Apr 2011 00:13:41 +1000
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
Message-ID: <BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>

On Sat, Apr 9, 2011 at 12:04 AM, Mike Graham <mikegraham at gmail.com> wrote:
> Neither of these seem justified for a rare case (this sort of patter
> is fairly rare, most notably this one example) when there's nothing
> that awful about the current solution.

Actually, there is an awful lot of code in the wild that gives
incorrect and wildly misleading error messages *because* correctly
checking the errno attribute is so rare. PEP 3151 would improve the
quality of information provided by a lot of tracebacks and error
messages without many of those developers needing to even set finger
to keyboard.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From mikegraham at gmail.com  Fri Apr  8 16:31:41 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Fri, 8 Apr 2011 10:31:41 -0400
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
Message-ID: <BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>

On Fri, Apr 8, 2011 at 10:13 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sat, Apr 9, 2011 at 12:04 AM, Mike Graham <mikegraham at gmail.com> wrote:
>> Neither of these seem justified for a rare case (this sort of patter
>> is fairly rare, most notably this one example) when there's nothing
>> that awful about the current solution.
>
> Actually, there is an awful lot of code in the wild that gives
> incorrect and wildly misleading error messages *because* correctly
> checking the errno attribute is so rare. PEP 3151 would improve the
> quality of information provided by a lot of tracebacks and error
> messages without many of those developers needing to even set finger
> to keyboard.

But Nick, that's different from what I'm saying is rare.

I'm saying that the situation where we need an if->raise on a constant
parameter is rare (this being almost the only common case). The issue
of whether people handle that case being rare is separate. Neither the
"except Foo as e if f(e):" syntax nor the "except foo(some_errno):"
pattern propose something that stops people from doing stupid stuff
like "except IOError" with no check.

Mike


From guido at python.org  Fri Apr  8 19:11:34 2011
From: guido at python.org (Guido van Rossum)
Date: Fri, 8 Apr 2011 10:11:34 -0700
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
Message-ID: <BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>

With apologies for not reading the PEP or this thread in full, some comments:

- I really like the syntax "except <exc> [as <var>] [if <test>]:".
This addresses a pretty common use case in my experience. I don't care
for the alternate proposed syntax that started this thread. I'm not
sure that the 'if' subclause makes sense without the 'as' subclause,
since most likely you'd want to refer to the caught exception. I note
that it is more powerful than putting "if not <test>: raise" in the
body of the except-clause, because it will allow subsequent except
clauses to match still. I also note that it is a much "cleaner" change
than (again) reorganizing the exception hierarchy, since there is no
backward compatibility to consider.

- Regarding restructuring the exception tree, I don't think this needs
to wait for Python 4. (We've done it several times during Python 2.)
But it is very tricky to do it without breaking existing code, and I'm
not sure that deprecations will help that much. E.g. if two exceptions
A and B are currently not related via inheritance, and we were to make
A a subclass of B, then in code of the form try: ... except B: ...
except A: ... the except A clause would become unreachable. Maybe it's
easier to propose small changes instead of trying to refactor a huge
section of the exception tree?

- Quite independently, I think it is time that we started documenting
which exceptions are raised from standard APIs. The trick is to avoid
overdocumenting: there is no point in documenting that everything can
raise MemoryError or that passing in wildly invalid arguments can
raise TypeError or AttributeError (and pretty much anything else). But
the variety of exceptions that may be raised by something like
urlopen, depending on which layer of the network and I/O stacks
detects a problem are worth documenting. Maybe this example could even
be a starting point for a rationalization of some of the exceptions
involved. (And if we find there are legitimate situations where a
networking or I/O problem raises a "generic" error such as TypeError
or AttributeError, those are probably bugs that should be fixed in the
code.)

- Lastly, there is a bunch of standard exception types that are
usually caused by bad code (as opposed to bad data or an environmental
problem). This would include NameError, TypeError, AttributeError, but
not KeyError, IndexError, ValueError. Perhaps it makes sense to
introduce a common base class for these "bad code" exceptions? Unlike
the other exception types, I think that the set of "bad code"
exceptions is pretty much static; a new library module is not going to
define raise a new kind of "bad code" exception. (But it may well
define a new kind of "bad data" exception, and I don't think we need a
common base class for all "bad data" or "bad state" exceptions.)

-- 
--Guido van Rossum (python.org/~guido)


From ericsnowcurrently at gmail.com  Fri Apr  8 19:31:43 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 8 Apr 2011 11:31:43 -0600
Subject: [Python-ideas] [Python-Ideas] AST Transformation Hooks for
	Domain Specific Languages
In-Reply-To: <BANLkTi=NcHXn0USiQkMMADHy2ZNLgDD3Gw@mail.gmail.com>
References: <BANLkTi=NcHXn0USiQkMMADHy2ZNLgDD3Gw@mail.gmail.com>
Message-ID: <BANLkTimJbdG5dW7FU5BkjvHU9wY7V1jD_g@mail.gmail.com>

On Fri, Apr 8, 2011 at 5:29 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> A few odds and ends from recent discussions finally clicked into
> something potentially interesting earlier this evening. Or possibly
> just something insane. I'm not quite decided on that point as yet (but
> leaning towards the latter).
>
> Anyway, without further ado, I present:
>
> AST Transformation Hooks for Domain Specific Languages
> ======================================================
>
> Consider:
>
> # In some other module
> ast.register_dsl("dsl.sql", dsl.sql.TransformAST)
>
> # In a module using that DSL
> import dsl.sql
> def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql:
>    select address
>    from people
>    where name = {name} and dob = {dob}
>
>
I like how you moved the from to after the parameter list.  It makes it less
complicated.  Annotations ( -> int: ) will still go after that, right?


>

Suppose that the standard AST for the latter looked something like:
>
>    DSL(syntax="dsl.sql",
>        name='lookup_address',
>        args=arguments(
>            args=[arg(arg='name',
>                      annotation=<Normal AST for "dsl.sql.char">),
>                  arg(arg='dob',
>                      annotation=<Normal AST for "dsl.sql.date">)],
>            vararg=None, varargannotation=None,
>            kwonlyargs=[], kwarg=None, kwargannotation=None,
>            defaults=[], kw_defaults=[]),
>        body=[Expr(value=Str(s='select address\nfrom people\nwhere
> name = {name} and dob = {dob}'))],
>        decorator_list=[],
>        returns=None)
>
> (For those not familiar with the AST, the above is actually just the
> existing Function node with a "syntax" attribute added)
>
> At *compile* time (note, *not* function definition time), the
> registered AST transformation hook would be invoked and would replace
> that DSL node with "standard" AST nodes.
>
>
So the AST transform would not necessarily return a Function node...


> For example, depending on the design of the DSL and its support code,
> the above example might be equivalent to:
>
>    @dsl.sql.escape_and_validate_args
>    def lookup_address(name: dsl.sql.char, dob: dsl.sql.date):
>       args = dict(name=name, dob=dob)
>       query = "select address\nfrom people\nwhere name = {name} and
> dob = {dob}"
>       return dsl.sql.cursor(query, args)
>
>
> As a simpler example, consider something like:
>
>    def f() from all_nonlocal:
>        x += 1
>        y -= 2
>
> That was translated at compile time into:
>
>    def f():
>        nonlocal x, y
>        x += 1
>        y -= 2
>
> My first pass at a rough protocol for the AST transformers suggests
> they would only need two methods:
>
>  get_cookie() - Magic cookie to add to PYC files containing instances
> of the DSL (allows recompilation to be forced if the DSL is updated)
>  transform_AST(node) - a DSL() node is passed in, expected to return
> an AST containing no DSL nodes (SyntaxError if one is found)
>
> Attempts to use an unregistered DSL would trigger SyntaxError
>
> So there you are, that's the crazy idea. The stoning of the heretic
> may now commence :)
>
> Where this idea came from was the various discussions about "make
> statement" style constructs and a conversation I had with Eric Snow at
> Pycon about function definition time really being *too late* to do
> anything particularly interesting that couldn't already be handled
> better in other ways. Some tricks Dave Malcolm had done to support
> Python level manipulation of the AST during compilation also played a
> big part, as did Eugene Toder's efforts to add an AST optimisation
> step to the compilation process.
>
>
I Like this idea better than the def-from with builders.  Since it is
compile-time, it certainly addresses some of the concerns that Guido brought
up, like run-time dynamic code generation and pyc files.  You simply have to
play within the confines of the AST, which seems like an effective
constraint.

Like Michael said, this would be great for trying out new language features.
  And like Eugene said, it may be worth doing in an import hook.  I've
actually been working on such a hook since the feedback I got for those
three ideas last week.  I doubt I will have anything right away, but I'll
let you know when I get there.  Regardless, I still like the idea of a
syntax that allows for the same freedom to define new stuff.

I wonder though if it would be subject to extensive abuse that would make
code harder to understand/use/debug.  I don't know that there are any other
language features that are extensively used that have his effect.  _maybe_
metaclasses, decorators, and the whole dot-lookup mechanism.  However, you
can wrap your head around those with a little explanation.  I guess this
would be similar.  It's good that the non-python is contained within the def
body.

I'm just thinking about what Raymond said last week about how all the
meta-programming stuff makes debugging harder.  I tend to agree.  I think of
the challenge of tracing through a problem I had in sqlalchemy a couple of
years ago that involved their use of metaclasses.

Anyway, love the idea.

-eric


> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110408/12dfcedb/attachment.html>

From mikegraham at gmail.com  Fri Apr  8 19:34:52 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Fri, 8 Apr 2011 13:34:52 -0400
Subject: [Python-ideas] Bad code exceptions (was PEP-3151 pattern-matching)
Message-ID: <BANLkTi=hYjm2JQLBXpzA5X20u=BTsYiWdQ@mail.gmail.com>

On Fri, Apr 8, 2011 at 1:11 PM, Guido van Rossum <guido at python.org> wrote:
> ...
>
> - Lastly, there is a bunch of standard exception types that are
> usually caused by bad code (as opposed to bad data or an environmental
> problem). This would include NameError, TypeError, AttributeError, but
> not KeyError, IndexError, ValueError. Perhaps it makes sense to
> introduce a common base class for these "bad code" exceptions? Unlike
> the other exception types, I think that the set of "bad code"
> exceptions is pretty much static; a new library module is not going to
> define raise a new kind of "bad code" exception. (But it may well
> define a new kind of "bad data" exception, and I don't think we need a
> common base class for all "bad data" or "bad state" exceptions.)
>
> --
> --Guido van Rossum (python.org/~guido)

The main usage I see for this is "ignore all exceptions except bad
programming exceptions". People try to do this now with a bare except
clause. 97% of the time, doing this is very ill-advised.

My gut reaction is that providing such a thing will encourage people
to use catchall exceptions. The penalty of making people write
"(KeyboardInterrupt, NameError, AttributeError)" when they want this
will be beneficial by a) not further complicating the exception
hierarchy, b) not encouraging catchalls, and c) making the developer
decide what they want to ignore and what they want to propagate for
borderline cases like TypeError.


From mikegraham at gmail.com  Fri Apr  8 19:52:18 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Fri, 8 Apr 2011 13:52:18 -0400
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
Message-ID: <BANLkTimVkUvx+SRJvjT3OADe6HehNpFSew@mail.gmail.com>

On Fri, Apr 8, 2011 at 1:11 PM, Guido van Rossum <guido at python.org> wrote:
> With apologies for not reading the PEP or this thread in full, some comments:
>
> - I really like the syntax "except <exc> [as <var>] [if <test>]:".
> This addresses a pretty common use case in my experience. I don't care
> for the alternate proposed syntax that started this thread. I'm not
> sure that the 'if' subclause makes sense without the 'as' subclause,
> since most likely you'd want to refer to the caught exception. I note
> that it is more powerful than putting "if not <test>: raise" in the
> body of the except-clause, because it will allow subsequent except
> clauses to match still. I also note that it is a much "cleaner" change
> than (again) reorganizing the exception hierarchy, since there is no
> backward compatibility to consider.

Of course, multiple cases can currently be handled by an if statement

except Foo as e:
    if e.bar == baz:
        ...
    elif e.bar == qux:
        ...
    else:
        raise

This code isn't so bad: it's not hard to understand, it's an existing
pattern, it uses general, powerful constructs.

I think the main opposition is "People aren't currently doing this!"
The problem is, introducing the "except...if" syntax doesn't actually
make anyone do it either. I understand that adding syntax can be seen
as a nudge, but I'm skeptical that it's a strong enough one to justify
the cost.

> - Quite independently, I think it is time that we started documenting
> which exceptions are raised from standard APIs. The trick is to avoid
> overdocumenting: there is no point in documenting that everything can
> raise MemoryError or that passing in wildly invalid arguments can
> raise TypeError or AttributeError (and pretty much anything else). But
> the variety of exceptions that may be raised by something like
> urlopen, depending on which layer of the network and I/O stacks
> detects a problem are worth documenting. Maybe this example could even
> be a starting point for a rationalization of some of the exceptions
> involved. (And if we find there are legitimate situations where a
> networking or I/O problem raises a "generic" error such as TypeError
> or AttributeError, those are probably bugs that should be fixed in the
> code.)

I couldn't agree more with this.


Mike


From guido at python.org  Fri Apr  8 19:52:16 2011
From: guido at python.org (Guido van Rossum)
Date: Fri, 8 Apr 2011 10:52:16 -0700
Subject: [Python-ideas] Bad code exceptions (was PEP-3151
	pattern-matching)
In-Reply-To: <BANLkTi=hYjm2JQLBXpzA5X20u=BTsYiWdQ@mail.gmail.com>
References: <BANLkTi=hYjm2JQLBXpzA5X20u=BTsYiWdQ@mail.gmail.com>
Message-ID: <BANLkTikNfKQB48UkX+j6j3RccAY4F1eCZQ@mail.gmail.com>

On Fri, Apr 8, 2011 at 10:34 AM, Mike Graham <mikegraham at gmail.com> wrote:
> On Fri, Apr 8, 2011 at 1:11 PM, Guido van Rossum <guido at python.org> wrote:
>> ...
>>
>> - Lastly, there is a bunch of standard exception types that are
>> usually caused by bad code (as opposed to bad data or an environmental
>> problem). This would include NameError, TypeError, AttributeError, but
>> not KeyError, IndexError, ValueError. Perhaps it makes sense to
>> introduce a common base class for these "bad code" exceptions? Unlike
>> the other exception types, I think that the set of "bad code"
>> exceptions is pretty much static; a new library module is not going to
>> define raise a new kind of "bad code" exception. (But it may well
>> define a new kind of "bad data" exception, and I don't think we need a
>> common base class for all "bad data" or "bad state" exceptions.)

> The main usage I see for this is "ignore all exceptions except bad
> programming exceptions". People try to do this now with a bare except
> clause. 97% of the time, doing this is very ill-advised.
>
> My gut reaction is that providing such a thing will encourage people
> to use catchall exceptions. The penalty of making people write
> "(KeyboardInterrupt, NameError, AttributeError)" when they want this
> will be beneficial by a) not further complicating the exception
> hierarchy, b) not encouraging catchalls, and c) making the developer
> decide what they want to ignore and what they want to propagate for
> borderline cases like TypeError.

I'm sorry, but I'm not quite following what you are saying here,
perhaps due to the double negatives you seem to be using. (Or perhaps
because my brain is not working 100% this morning. :-)

Could you elaborate, perhaps with a few examples of good practice, bad
practice, how each would be written with and without the hypothetical
"BadCodeError" exception, and which forms you'd consider good and bad
style?

I could also see how combining this with the 'if' subclause might
actually be beneficial:

try:
  <something highly experimental>
except Exception as e if not isinstance(e, BadCodeError):
  <apparently there was some bad data>

Of course in reality bad data will often lead to a BadCodeError, the
most common example being trying to call a method on a value that is
unexpectedly None. Maybe getattr(None, anything) should raise some
other exception? (Clearly just thinking aloud here. :-)

-- 
--Guido van Rossum (python.org/~guido)


From guido at python.org  Fri Apr  8 19:56:50 2011
From: guido at python.org (Guido van Rossum)
Date: Fri, 8 Apr 2011 10:56:50 -0700
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTimVkUvx+SRJvjT3OADe6HehNpFSew@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
	<BANLkTimVkUvx+SRJvjT3OADe6HehNpFSew@mail.gmail.com>
Message-ID: <BANLkTi=nXwgPRoVtfduL0_sMB+tBMH+Swg@mail.gmail.com>

On Fri, Apr 8, 2011 at 10:52 AM, Mike Graham <mikegraham at gmail.com> wrote:
> On Fri, Apr 8, 2011 at 1:11 PM, Guido van Rossum <guido at python.org> wrote:
>> With apologies for not reading the PEP or this thread in full, some comments:
>>
>> - I really like the syntax "except <exc> [as <var>] [if <test>]:".
>> This addresses a pretty common use case in my experience. I don't care
>> for the alternate proposed syntax that started this thread. I'm not
>> sure that the 'if' subclause makes sense without the 'as' subclause,
>> since most likely you'd want to refer to the caught exception. I note
>> that it is more powerful than putting "if not <test>: raise" in the
>> body of the except-clause, because it will allow subsequent except
>> clauses to match still. I also note that it is a much "cleaner" change
>> than (again) reorganizing the exception hierarchy, since there is no
>> backward compatibility to consider.
>
> Of course, multiple cases can currently be handled by an if statement
>
> except Foo as e:
> ? ?if e.bar == baz:
> ? ? ? ?...
> ? ?elif e.bar == qux:
> ? ? ? ?...
> ? ?else:
> ? ? ? ?raise
>
> This code isn't so bad: it's not hard to understand, it's an existing
> pattern, it uses general, powerful constructs.

The problem (as I tried to say, but apparently not clearly enough) is
that if there's a later more general except clause on the same block
it won't work. E.g.

except Foo as e:
  <code as above>
except Exception:
  <log traceback>

The 'raise' in your code above does not transfer control to the
"except Exception:" clause. The current solution is either to have two
nested try/except blocks, or duplicate the logging code; neither is
great.

> I think the main opposition is "People aren't currently doing this!"

People aren't currently doing what? I have seen many examples like
that, and have myself encountered several times the problem I
explained just now.

> The problem is, introducing the "except...if" syntax doesn't actually
> make anyone do it either. I understand that adding syntax can be seen
> as a nudge, but I'm skeptical that it's a strong enough one to justify
> the cost.

What cost? Adding new optional syntax that doesn't introduce new
keywords is actually pretty cheap -- certainly cheaper than
restructuring the exception hierarchy.

-- 
--Guido van Rossum (python.org/~guido)


From solipsis at pitrou.net  Fri Apr  8 20:18:07 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 8 Apr 2011 20:18:07 +0200
Subject: [Python-ideas] PEP-3151 pattern-matching
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
Message-ID: <20110408201807.6dc58ebf@pitrou.net>

On Fri, 8 Apr 2011 10:11:34 -0700
Guido van Rossum <guido at python.org> wrote:
> With apologies for not reading the PEP or this thread in full, some comments:
> 
> - I really like the syntax "except <exc> [as <var>] [if <test>]:".
> This addresses a pretty common use case in my experience. I don't care
> for the alternate proposed syntax that started this thread. I'm not
> sure that the 'if' subclause makes sense without the 'as' subclause,
> since most likely you'd want to refer to the caught exception. I note
> that it is more powerful than putting "if not <test>: raise" in the
> body of the except-clause, because it will allow subsequent except
> clauses to match still. I also note that it is a much "cleaner" change
> than (again) reorganizing the exception hierarchy, since there is no
> backward compatibility to consider.

My main issue with said new syntax is that it doesn't make things much
easier to write. You still have to type an explicit condition, and
remember the appropriate errno mnemonic for the situation. The very
notion of "errno" and its plethora of abbreviated error codes is ok for
people used to C programming, but certainly alien to other programmers
(how long does it take to remember that "file not found" is spelled
"ENOENT"?).

The fact that exception messages typically indicate the errno *number*,
not mnemonic, is an additional annoyance (errno numbers are not
standardized and can vary from system to system).

> - Regarding restructuring the exception tree, I don't think this needs
> to wait for Python 4. (We've done it several times during Python 2.)
> But it is very tricky to do it without breaking existing code, and I'm
> not sure that deprecations will help that much. E.g. if two exceptions
> A and B are currently not related via inheritance, and we were to make
> A a subclass of B, then in code of the form try: ... except B: ...
> except A: ... the except A clause would become unreachable. Maybe it's
> easier to propose small changes instead of trying to refactor a huge
> section of the exception tree?

Well, I fear that small changes will not lead us very far. We have to
level the existing tree before adding new types, because otherwise it's
not obvious where these new types should be grafted, since in practice
OSError, IOError or socket.error can be used for very similar error
conditions.

> - Quite independently, I think it is time that we started documenting
> which exceptions are raised from standard APIs.

Agreed, and unit-test them too.

> (But it may well
> define a new kind of "bad data" exception, and I don't think we need a
> common base class for all "bad data" or "bad state" exceptions.)

Isn't it ValueError actually? For example, closed files raise
ValueError when you try to do an operation on them.

Regards

Antoine.




From mikegraham at gmail.com  Fri Apr  8 20:22:01 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Fri, 8 Apr 2011 14:22:01 -0400
Subject: [Python-ideas] Bad code exceptions (was PEP-3151
	pattern-matching)
In-Reply-To: <BANLkTikNfKQB48UkX+j6j3RccAY4F1eCZQ@mail.gmail.com>
References: <BANLkTi=hYjm2JQLBXpzA5X20u=BTsYiWdQ@mail.gmail.com>
	<BANLkTikNfKQB48UkX+j6j3RccAY4F1eCZQ@mail.gmail.com>
Message-ID: <BANLkTi=EeokjNY7b5f9d=2URpMOWoji7Wg@mail.gmail.com>

On Fri, Apr 8, 2011 at 1:52 PM, Guido van Rossum <guido at python.org> wrote:
> I'm sorry, but I'm not quite following what you are saying here,
> perhaps due to the double negatives you seem to be using. (Or perhaps
> because my brain is not working 100% this morning. :-)

I'm pretty sure it's me!

> Could you elaborate, perhaps with a few examples of good practice, bad
> practice, how each would be written with and without the hypothetical
> "BadCodeError" exception, and which forms you'd consider good and bad
> style?

I imagine common usage would be

try:
    website = open_and_parse(url)
except Exception as e if not isinstance(e, BadCodeError):
    print("Error downloading %s." % url)

because I currently see tons of

try:
    website = open_and_parse(url)
except:
    print("Error downloading %s." % url)

The right thing to do would likely be to figure out which errors mean
you couldn't download the website and catch those. Here you might
accidentally squash exceptions raised in parsing and issue the wrong
error message.


A more appropriate usage might be

def my_event_loop(self):
    while True:
        try:
            self.advance()
        except BaseException as e:
             self.logger.exception()
             if isinstance(e, (NameError, AttributeError,
                               KeyboardInterrupt, SystemExit,
                               MemoryError)):
                  raise

Catchall exceptions (even of a slightly more limited nature) should be
a rarity and typically is caused by bugs. Since this is something that
should seldom be done and since there are borderline cases, I don't
think that a language-level change helps.

This list sort of suggests to me that I'm thinking more of
DoNotCatchThisError, which includes BadCodeError and
SomeExternalThingICannotHelpError.

> I could also see how combining this with the 'if' subclause might
> actually be beneficial:
>
> try:
> ?<something highly experimental>
> except Exception as e if not isinstance(e, BadCodeError):
> ?<apparently there was some bad data>

I imagine this would be by far the most common use of BadCodeError,
which suggests to me that if we had something like this, it would be
important to have (a better-named GoodCodeError) or CatchableError.

> Of course in reality bad data will often lead to a BadCodeError, the
> most common example being trying to call a method on a value that is
> unexpectedly None. Maybe getattr(None, anything) should raise some
> other exception? (Clearly just thinking aloud here. :-)
>
> --
> --Guido van Rossum (python.org/~guido)

Mike


From debatem1 at gmail.com  Fri Apr  8 20:24:05 2011
From: debatem1 at gmail.com (geremy condra)
Date: Fri, 8 Apr 2011 11:24:05 -0700
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
Message-ID: <BANLkTi=COQzetiMzYMnHsFW_fNgwnCuc7g@mail.gmail.com>

On Fri, Apr 8, 2011 at 10:11 AM, Guido van Rossum <guido at python.org> wrote:

<snip>

> - Quite independently, I think it is time that we started documenting
> which exceptions are raised from standard APIs. The trick is to avoid
> overdocumenting: there is no point in documenting that everything can
> raise MemoryError or that passing in wildly invalid arguments can
> raise TypeError or AttributeError (and pretty much anything else). But
> the variety of exceptions that may be raised by something like
> urlopen, depending on which layer of the network and I/O stacks
> detects a problem are worth documenting. Maybe this example could even
> be a starting point for a rationalization of some of the exceptions
> involved. (And if we find there are legitimate situations where a
> networking or I/O problem raises a "generic" error such as TypeError
> or AttributeError, those are probably bugs that should be fixed in the
> code.)

This would be very nice.

> - Lastly, there is a bunch of standard exception types that are
> usually caused by bad code (as opposed to bad data or an environmental
> problem). This would include NameError, TypeError, AttributeError, but
> not KeyError, IndexError, ValueError. Perhaps it makes sense to
> introduce a common base class for these "bad code" exceptions? Unlike
> the other exception types, I think that the set of "bad code"
> exceptions is pretty much static; a new library module is not going to
> define raise a new kind of "bad code" exception. (But it may well
> define a new kind of "bad data" exception, and I don't think we need a
> common base class for all "bad data" or "bad state" exceptions.)

I actually like both the BadCode exception and the BadData exception.

Geremy Condra


From guido at python.org  Fri Apr  8 20:32:39 2011
From: guido at python.org (Guido van Rossum)
Date: Fri, 8 Apr 2011 11:32:39 -0700
Subject: [Python-ideas] Bad code exceptions (was PEP-3151
	pattern-matching)
In-Reply-To: <BANLkTi=EeokjNY7b5f9d=2URpMOWoji7Wg@mail.gmail.com>
References: <BANLkTi=hYjm2JQLBXpzA5X20u=BTsYiWdQ@mail.gmail.com>
	<BANLkTikNfKQB48UkX+j6j3RccAY4F1eCZQ@mail.gmail.com>
	<BANLkTi=EeokjNY7b5f9d=2URpMOWoji7Wg@mail.gmail.com>
Message-ID: <BANLkTik93Bc9voL8ZyXUn6anUcaeOFKhvw@mail.gmail.com>

On Fri, Apr 8, 2011 at 11:22 AM, Mike Graham <mikegraham at gmail.com> wrote:
> On Fri, Apr 8, 2011 at 1:52 PM, Guido van Rossum <guido at python.org> wrote:
>> Could you elaborate, perhaps with a few examples of good practice, bad
>> practice, how each would be written with and without the hypothetical
>> "BadCodeError" exception, and which forms you'd consider good and bad
>> style?
>
> I imagine common usage would be
>
> try:
> ? ?website = open_and_parse(url)
> except Exception as e if not isinstance(e, BadCodeError):
> ? ?print("Error downloading %s." % url)
>
> because I currently see tons of
>
> try:
> ? ?website = open_and_parse(url)
> except:
> ? ?print("Error downloading %s." % url)
>
> The right thing to do would likely be to figure out which errors mean
> you couldn't download the website and catch those. Here you might
> accidentally squash exceptions raised in parsing and issue the wrong
> error message.

Gotcha. I would actually say this is somewhat special because urlopen
can raise so many different errors; thatr's why I called it out as an
example earlier. If all errors raisable by urlopen due to external
circumstances would derive from some common base exception, people
would catch that exception. I agree that the "except Exception as e if
not isinstance(BadCodeError):" idiom is not great, and given that it
is such a mouthful, people who are currently writing "except:" are
unlikely to change their ways.

> A more appropriate usage might be
>
> def my_event_loop(self):
> ? ?while True:
> ? ? ? ?try:
> ? ? ? ? ? ?self.advance()
> ? ? ? ?except BaseException as e:
> ? ? ? ? ? ? self.logger.exception()
> ? ? ? ? ? ? if isinstance(e, (NameError, AttributeError,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? KeyboardInterrupt, SystemExit,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? MemoryError)):

You seem to be missing the reason why we introduced BaseException
here... Read PEP 352.

> ? ? ? ? ? ? ? ? ?raise
>
> Catchall exceptions (even of a slightly more limited nature) should be
> a rarity and typically is caused by bugs. Since this is something that
> should seldom be done and since there are borderline cases, I don't
> think that a language-level change helps.

But honestly that wasn't my intended use case. I was thinking more
along the lines of trying to catch OSError with a specific errno
value:

except OSError as e if e.errno == errno.ENOTOBACCO:
  logging.info('Pipe has no tobacco')

> This list sort of suggests to me that I'm thinking more of
> DoNotCatchThisError, which includes BadCodeError and
> SomeExternalThingICannotHelpError.

I don't think we can define "negative" exception classes.

>> I could also see how combining this with the 'if' subclause might
>> actually be beneficial:
>>
>> try:
>> ?<something highly experimental>
>> except Exception as e if not isinstance(e, BadCodeError):
>> ?<apparently there was some bad data>
>
> I imagine this would be by far the most common use of BadCodeError,
> which suggests to me that if we had something like this, it would be
> important to have (a better-named GoodCodeError) or CatchableError.

But it's too late for that -- BadCodeError must derive from Exception,
and most 3rd party code also defines exceptions deriving from
Exception.

-- 
--Guido van Rossum (python.org/~guido)


From jeanpierreda at gmail.com  Fri Apr  8 21:00:50 2011
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Fri, 8 Apr 2011 15:00:50 -0400
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <20110408201807.6dc58ebf@pitrou.net>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
	<20110408201807.6dc58ebf@pitrou.net>
Message-ID: <BANLkTik680zMecEcZCZPJt6BDrLAgOWrpg@mail.gmail.com>

On Fri, Apr 8, 2011 at 2:18 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Fri, 8 Apr 2011 10:11:34 -0700
> Guido van Rossum <guido at python.org> wrote:
>> With apologies for not reading the PEP or this thread in full, some comments:
>>
>> - I really like the syntax "except <exc> [as <var>] [if <test>]:".
>> This addresses a pretty common use case in my experience. I don't care
>> for the alternate proposed syntax that started this thread. I'm not
>> sure that the 'if' subclause makes sense without the 'as' subclause,
>> since most likely you'd want to refer to the caught exception. I note
>> that it is more powerful than putting "if not <test>: raise" in the
>> body of the except-clause, because it will allow subsequent except
>> clauses to match still. I also note that it is a much "cleaner" change
>> than (again) reorganizing the exception hierarchy, since there is no
>> backward compatibility to consider.
>
> My main issue with said new syntax is that it doesn't make things much
> easier to write. You still have to type an explicit condition, and
> remember the appropriate errno mnemonic for the situation. The very
> notion of "errno" and its plethora of abbreviated error codes is ok for
> people used to C programming, but certainly alien to other programmers
> (how long does it take to remember that "file not found" is spelled
> "ENOENT"?).
>
> The fact that exception messages typically indicate the errno *number*,
> not mnemonic, is an additional annoyance (errno numbers are not
> standardized and can vary from system to system).

It wouldn't be that difficult to customize the IOError and OSError
reprs to display the errno mnemonic instead of a number, if that's an
issue.

I don't think it's fair to characterize errnos as only for C
programmers. Some other language use errno mnemonics too, not just C
and Python. It's a cross-language way of understanding certain kinds
of error conditions, and as such it makes it a bit easier to do
things. What this PEP does is introduce a language-specific exception
hierarchy instead. I think it's a good idea, but it's not as clearly
superior to errnos as you suggest. It's easier to remember the
language-specific exception hierarchy if you don't have to deal with
other languages that use errnos, but if you do, the situation is
different.

>> - Regarding restructuring the exception tree, I don't think this needs
>> to wait for Python 4. (We've done it several times during Python 2.)
>> But it is very tricky to do it without breaking existing code, and I'm
>> not sure that deprecations will help that much. E.g. if two exceptions
>> A and B are currently not related via inheritance, and we were to make
>> A a subclass of B, then in code of the form try: ... except B: ...
>> except A: ... the except A clause would become unreachable. Maybe it's
>> easier to propose small changes instead of trying to refactor a huge
>> section of the exception tree?
>
> Well, I fear that small changes will not lead us very far. We have to
> level the existing tree before adding new types, because otherwise it's
> not obvious where these new types should be grafted, since in practice
> OSError, IOError or socket.error can be used for very similar error
> conditions.
>
>> - Quite independently, I think it is time that we started documenting
>> which exceptions are raised from standard APIs.
>
> Agreed, and unit-test them too.
>
>> (But it may well
>> define a new kind of "bad data" exception, and I don't think we need a
>> common base class for all "bad data" or "bad state" exceptions.)
>
> Isn't it ValueError actually? For example, closed files raise
> ValueError when you try to do an operation on them.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From solipsis at pitrou.net  Fri Apr  8 21:06:50 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 8 Apr 2011 21:06:50 +0200
Subject: [Python-ideas] PEP-3151 pattern-matching
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
	<20110408201807.6dc58ebf@pitrou.net>
	<BANLkTik680zMecEcZCZPJt6BDrLAgOWrpg@mail.gmail.com>
Message-ID: <20110408210650.24b29d26@pitrou.net>

On Fri, 8 Apr 2011 15:00:50 -0400
Devin Jeanpierre <jeanpierreda at gmail.com>
wrote:
> What this PEP does is introduce a language-specific exception
> hierarchy instead. I think it's a good idea, but it's not as clearly
> superior to errnos as you suggest. It's easier to remember the
> language-specific exception hierarchy if you don't have to deal with
> other languages that use errnos, but if you do, the situation is
> different.

True, but that's like saying we should use sprintf() for string
formatting and fgets() for file access since other languages such as C
and PHP use them. In trying to find elegant solutions to problems, we
often diverge from solutions common in other languages.

(actually, if we followed the errno paradigm closely, we shouldn't even
raise exceptions, but return the error code or store it in a
thread-local variable instead ;-))

Regards

Antoine.




From solipsis at pitrou.net  Fri Apr  8 21:09:29 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 8 Apr 2011 21:09:29 +0200
Subject: [Python-ideas] Bad code exceptions (was PEP-3151
	pattern-matching)
References: <BANLkTi=hYjm2JQLBXpzA5X20u=BTsYiWdQ@mail.gmail.com>
	<BANLkTikNfKQB48UkX+j6j3RccAY4F1eCZQ@mail.gmail.com>
	<BANLkTi=EeokjNY7b5f9d=2URpMOWoji7Wg@mail.gmail.com>
	<BANLkTik93Bc9voL8ZyXUn6anUcaeOFKhvw@mail.gmail.com>
Message-ID: <20110408210929.1190c6b4@pitrou.net>

On Fri, 8 Apr 2011 11:32:39 -0700
Guido van Rossum <guido at python.org> wrote:
> 
> But honestly that wasn't my intended use case. I was thinking more
> along the lines of trying to catch OSError with a specific errno
> value:
> 
> except OSError as e if e.errno == errno.ENOTOBACCO:
>   logging.info('Pipe has no tobacco')

Given errno naming conventions, this should probably be errno.ENOTOB ;)

Regards

Antoine.




From mikegraham at gmail.com  Fri Apr  8 21:15:24 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Fri, 8 Apr 2011 15:15:24 -0400
Subject: [Python-ideas] Bad code exceptions (was PEP-3151
	pattern-matching)
In-Reply-To: <BANLkTik93Bc9voL8ZyXUn6anUcaeOFKhvw@mail.gmail.com>
References: <BANLkTi=hYjm2JQLBXpzA5X20u=BTsYiWdQ@mail.gmail.com>
	<BANLkTikNfKQB48UkX+j6j3RccAY4F1eCZQ@mail.gmail.com>
	<BANLkTi=EeokjNY7b5f9d=2URpMOWoji7Wg@mail.gmail.com>
	<BANLkTik93Bc9voL8ZyXUn6anUcaeOFKhvw@mail.gmail.com>
Message-ID: <BANLkTin5ZPTarF-6jp6q3uJNrnW4LzAV_g@mail.gmail.com>

On Fri, Apr 8, 2011 at 2:32 PM, Guido van Rossum <guido at python.org> wrote:
> On Fri, Apr 8, 2011 at 11:22 AM, Mike Graham <mikegraham at gmail.com> wrote:
>> def my_event_loop(self):
>>    while True:
>>        try:
>>            self.advance()
>>        except BaseException as e:
>>             self.logger.exception()
>>             if isinstance(e, (NameError, AttributeError,
>>                               KeyboardInterrupt, SystemExit,
>>                               MemoryError)):
>
> You seem to be missing the reason why we introduced BaseException
> here... Read PEP 352.

I think I get what BaseException was conceived for, but it usually
proves unsuitable for my usage. If I just do "except Exception" in
this code (is that the idea?), it doesn't behave like I'd want:
a) If my program exits due to a KeyboardInterrupt, it doesn't make it
into my log, which I may want.
b) If my program exits due to a loose GeneratorExit, it doesn't make
it into my log, which I almost certainly want, and
c) Plenty of Exception subclasses (like MemoryError and any sort of
candidate for BadCodeError) are things that should be treated as
uncatchable as often as the exceptions which don't subclass Exception.

Perhaps I'm letting this cloud my vision of why you want BadCodeError.

>> Catchall exceptions (even of a slightly more limited nature) should be
>> a rarity and typically is caused by bugs. Since this is something that
>> should seldom be done and since there are borderline cases, I don't
>> think that a language-level change helps.
>
> But honestly that wasn't my intended use case. I was thinking more
> along the lines of trying to catch OSError with a specific errno
> value:
>
> except OSError as e if e.errno == errno.ENOTOBACCO:
>  logging.info('Pipe has no tobacco')

I'm confused how that relates to BadeCodeError?

> I don't think we can define "negative" exception classes.

class NotBadCodeError(Exception):
    __metaclass__ = abc.ABCMeta

    @classmethod
    def __subclasshook__(self, potential_subclass):
        if not issubclass(potential_subclass, BadCodeError):
            return True
        return False

DON'T JUDGE ME!

> But it's too late for that -- BadCodeError must derive from Exception,
> and most 3rd party code also defines exceptions deriving from
> Exception.
Right.


Mike


From jeanpierreda at gmail.com  Fri Apr  8 23:14:23 2011
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Fri, 8 Apr 2011 17:14:23 -0400
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <20110408210650.24b29d26@pitrou.net>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
	<20110408201807.6dc58ebf@pitrou.net>
	<BANLkTik680zMecEcZCZPJt6BDrLAgOWrpg@mail.gmail.com>
	<20110408210650.24b29d26@pitrou.net>
Message-ID: <BANLkTi=bqTsGR0+x3v3M1wNzCbG4Tn+a2Q@mail.gmail.com>

On Fri, Apr 8, 2011 at 3:06 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> True, but that's like saying we should use sprintf() for string
> formatting and fgets() for file access since other languages such as C
> and PHP use them. In trying to find elegant solutions to problems, we
> often diverge from solutions common in other languages.
>
> (actually, if we followed the errno paradigm closely, we shouldn't even
> raise exceptions, but return the error code or store it in a
> thread-local variable instead ;-))
>
> Regards
>
> Antoine.

Well, not quite. The only case that I can see where errnos are
obviously bad is with multiple errnos that mean the same thing in
practice (so much so that they might even have the same value), making
proper IO handling inconvenient and non-obvious. In particular I'm
thinking of errors like EAGAIN and EWOULDBLOCK. The PEP's suggested
exception hierarchy, and its collapse of some errnos into the same
exception, is nice on that front (for example, BlockingIOError).
However, I interpret what you say as a complaint that errnos are
obscure and thus we should avoid making programmers memorize them, and
avoid encouraging their use. I don't agree with that. They aren't
obscure, they're a legitimate part of POSIX and fairly well-known.
Beginners to IO will have to memorize something anyway, and it may as
well be errnos. A more experienced programmer might come from from C
or C# or Ruby, or whatever, and would get less educational overhead
when doing Python IO, because all these languages share a little bit
of POSIX in their IO APIs.

It's not even about following it closely, but about building on what
people know. Python doesn't support sprintf, but it does (or perhaps
"did") support the sprintf syntax. It didn't follow it *closely* --
"%s" % 2 doesn't result in a segfault or anything -- but closely
enough so that programmers familiar with sprintf could get started
quickly, and programmers familiar with Python could understand
sprintf. The replacement, str.format, doesn't come from nowhere
either, but builds on a different string formatting tradition also
shared by C#. Having these ties to the outside world is useful and
desirable, it makes everyone's life easier. I think that breaking
these ties needs an overriding reason, generally relating to the
outside world actually hurting people. The PEP solves the issue where
people don't necessarily handle all the right errnos, without breaking
ties to the IO community and eliminating them altogether. That's
pretty cool. It also offers a way to make errno handling easier.
That's also pretty cool.

Devin


From guido at python.org  Fri Apr  8 23:50:02 2011
From: guido at python.org (Guido van Rossum)
Date: Fri, 8 Apr 2011 14:50:02 -0700
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <20110408201807.6dc58ebf@pitrou.net>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
	<20110408201807.6dc58ebf@pitrou.net>
Message-ID: <BANLkTikckxai4r5rQe0RCW30Z5+XrSEibg@mail.gmail.com>

On Fri, Apr 8, 2011 at 11:18 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Fri, 8 Apr 2011 10:11:34 -0700
> Guido van Rossum <guido at python.org> wrote:
>> With apologies for not reading the PEP or this thread in full, some comments:
>>
>> - I really like the syntax "except <exc> [as <var>] [if <test>]:".
>> This addresses a pretty common use case in my experience. I don't care
>> for the alternate proposed syntax that started this thread. I'm not
>> sure that the 'if' subclause makes sense without the 'as' subclause,
>> since most likely you'd want to refer to the caught exception. I note
>> that it is more powerful than putting "if not <test>: raise" in the
>> body of the except-clause, because it will allow subsequent except
>> clauses to match still. I also note that it is a much "cleaner" change
>> than (again) reorganizing the exception hierarchy, since there is no
>> backward compatibility to consider.
>
> My main issue with said new syntax is that it doesn't make things much
> easier to write.

As I explained in other messages, it also adds semantics that are not
so easily emulated with the existing syntax (you'd have to repeat code
or use nested try/except blocks).

> You still have to type an explicit condition, and
> remember the appropriate errno mnemonic for the situation. The very
> notion of "errno" and its plethora of abbreviated error codes is ok for
> people used to C programming, but certainly alien to other programmers
> (how long does it take to remember that "file not found" is spelled
> "ENOENT"?).

Well, this is a fact of life. There are hundreds of errno codes and
they vary across systems, both in names and in numbers (even if there
is a handful that exist nearly everywhere and almost always have the
same values). This is the nature of system calls, and some operating
systems just have a richer set of error conditions that they present
to the (advanced) programmer.

If you are careful you can probably come up with a few handfuls of
common error conditions for which it makes sense to introduce new
exceptions that are defined (and presumably raised) on all platforms.
"File not found" is one of those. But there are always error
conditions that are not in this list, and we would still need the
errno attribute and the errno module to map between numbers, names and
messages in the general case. And occasionally some programmer needs
to select behaviors depending on the precise errno value.

> The fact that exception messages typically indicate the errno *number*,
> not mnemonic, is an additional annoyance (errno numbers are not
> standardized and can vary from system to system).

That would seem to be a problem with the str() and/or repr() of
OSError, and perhaps we can fix it there.

>> - Regarding restructuring the exception tree, I don't think this needs
>> to wait for Python 4. (We've done it several times during Python 2.)
>> But it is very tricky to do it without breaking existing code, and I'm
>> not sure that deprecations will help that much. E.g. if two exceptions
>> A and B are currently not related via inheritance, and we were to make
>> A a subclass of B, then in code of the form try: ... except B: ...
>> except A: ... the except A clause would become unreachable. Maybe it's
>> easier to propose small changes instead of trying to refactor a huge
>> section of the exception tree?
>
> Well, I fear that small changes will not lead us very far. We have to
> level the existing tree before adding new types, because otherwise it's
> not obvious where these new types should be grafted, since in practice
> OSError, IOError or socket.error can be used for very similar error
> conditions.

Alas, you are right, they are all essentially the same thing but
raised by different operations (sometimes it's even the same
underlying system call, e.g. os.write vs. stream.write).

>> - Quite independently, I think it is time that we started documenting
>> which exceptions are raised from standard APIs.
>
> Agreed, and unit-test them too.
>
>> (But it may well
>> define a new kind of "bad data" exception, and I don't think we need a
>> common base class for all "bad data" or "bad state" exceptions.)
>
> Isn't it ValueError actually? For example, closed files raise
> ValueError when you try to do an operation on them.

No, there are lots of other exceptions indicating "bad arguments" or
"bad data" or "bad state" that don't derive from ValueError, from
KeyError to binhex.Error (a random example I picked from the stdlib).

-- 
--Guido van Rossum (python.org/~guido)


From solipsis at pitrou.net  Sat Apr  9 00:39:50 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 9 Apr 2011 00:39:50 +0200
Subject: [Python-ideas] PEP-3151 pattern-matching
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
	<20110408201807.6dc58ebf@pitrou.net>
	<BANLkTikckxai4r5rQe0RCW30Z5+XrSEibg@mail.gmail.com>
Message-ID: <20110409003950.6cb3eada@pitrou.net>

On Fri, 8 Apr 2011 14:50:02 -0700
Guido van Rossum <guido at python.org> wrote:
> 
> > You still have to type an explicit condition, and
> > remember the appropriate errno mnemonic for the situation. The very
> > notion of "errno" and its plethora of abbreviated error codes is ok for
> > people used to C programming, but certainly alien to other programmers
> > (how long does it take to remember that "file not found" is spelled
> > "ENOENT"?).
> 
> Well, this is a fact of life. There are hundreds of errno codes and
> they vary across systems, both in names and in numbers (even if there
> is a handful that exist nearly everywhere and almost always have the
> same values). This is the nature of system calls, and some operating
> systems just have a richer set of error conditions that they present
> to the (advanced) programmer.

Well, errnos are only the POSIX API side of system calls.
For example, a seasoned Windows C programmer might not have faced a
single EBADF or ENOENT in their entire career, since the Win32 API
doesn't use those.
Similarly, java.io.IOException doesn't seem to have any errno-alike
member, but it does have subclasses named e.g. FileNotFoundException.
(http://download.oracle.com/javase/1.5.0/docs/api/java/io/IOException.html)
Looking at .Net, System.IO.IOException does have a numerical "HResult"
member, but it doesn't seem related to POSIX errnos (?HRESULT is a
32-bit value, divided into three different fields: a severity code, a
facility code, and an error code?). It also has a couple of subclasses
such as FileNotFoundException.
(http://msdn.microsoft.com/en-us/library/system.io.ioexception.aspx)

> > The fact that exception messages typically indicate the errno *number*,
> > not mnemonic, is an additional annoyance (errno numbers are not
> > standardized and can vary from system to system).
> 
> That would seem to be a problem with the str() and/or repr() of
> OSError, and perhaps we can fix it there.

True.

Regards

Antoine.




From ericsnowcurrently at gmail.com  Sat Apr  9 00:56:54 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 8 Apr 2011 16:56:54 -0600
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTi=bqTsGR0+x3v3M1wNzCbG4Tn+a2Q@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
	<20110408201807.6dc58ebf@pitrou.net>
	<BANLkTik680zMecEcZCZPJt6BDrLAgOWrpg@mail.gmail.com>
	<20110408210650.24b29d26@pitrou.net>
	<BANLkTi=bqTsGR0+x3v3M1wNzCbG4Tn+a2Q@mail.gmail.com>
Message-ID: <BANLkTim8Z11Xm5jcKDhvkFAb-K3GZeUjjQ@mail.gmail.com>

On Fri, Apr 8, 2011 at 3:14 PM, Devin Jeanpierre <jeanpierreda at gmail.com>wrote:

> On Fri, Apr 8, 2011 at 3:06 PM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> > True, but that's like saying we should use sprintf() for string
> > formatting and fgets() for file access since other languages such as C
> > and PHP use them. In trying to find elegant solutions to problems, we
> > often diverge from solutions common in other languages.
> >
> > (actually, if we followed the errno paradigm closely, we shouldn't even
> > raise exceptions, but return the error code or store it in a
> > thread-local variable instead ;-))
> >
> > Regards
> >
> > Antoine.
>
> Well, not quite. The only case that I can see where errnos are
> obviously bad is with multiple errnos that mean the same thing in
> practice (so much so that they might even have the same value), making
> proper IO handling inconvenient and non-obvious. In particular I'm
> thinking of errors like EAGAIN and EWOULDBLOCK. The PEP's suggested
> exception hierarchy, and its collapse of some errnos into the same
> exception, is nice on that front (for example, BlockingIOError).
> However, I interpret what you say as a complaint that errnos are
> obscure and thus we should avoid making programmers memorize them, and
> avoid encouraging their use. I don't agree with that. They aren't
> obscure, they're a legitimate part of POSIX and fairly well-known.
> Beginners to IO will have to memorize something anyway, and it may as
> well be errnos.


I see your point about that value of knowing stuff, but why should I need to
learn errnos until I need them?  I deal with IO errors on occasion and end
up having to look it up each time.  That's a pain.

Consider how much IO is used in Python and how often people have to deal
with IO exceptions.  Why should we have to deal with errnos if there is a
way that fits into the exception hierarchy?  It's not that they are obscure.
 It's that we don't need the actual numbers or mnemonics  in our Python
code.  We can hide them behind more meaningful exceptions.  At least, that's
my take.

FYI, the the exception objects will still have the errno attribute that you
would find in the plain IOError:

http://www.python.org/dev/peps/pep-3151/#id26

-eric


> A more experienced programmer might come from from C
> or C# or Ruby, or whatever, and would get less educational overhead
> when doing Python IO, because all these languages share a little bit
> of POSIX in their IO APIs.
>
> It's not even about following it closely, but about building on what
> people know. Python doesn't support sprintf, but it does (or perhaps
> "did") support the sprintf syntax. It didn't follow it *closely* --
> "%s" % 2 doesn't result in a segfault or anything -- but closely
> enough so that programmers familiar with sprintf could get started
> quickly, and programmers familiar with Python could understand
> sprintf. The replacement, str.format, doesn't come from nowhere
> either, but builds on a different string formatting tradition also
> shared by C#. Having these ties to the outside world is useful and
> desirable, it makes everyone's life easier. I think that breaking
> these ties needs an overriding reason, generally relating to the
> outside world actually hurting people. The PEP solves the issue where
> people don't necessarily handle all the right errnos, without breaking
> ties to the IO community and eliminating them altogether. That's
> pretty cool. It also offers a way to make errno handling easier.
> That's also pretty cool.
>
> Devin
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110408/9f5b5de3/attachment.html>

From guido at python.org  Sat Apr  9 00:59:16 2011
From: guido at python.org (Guido van Rossum)
Date: Fri, 8 Apr 2011 15:59:16 -0700
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTim8Z11Xm5jcKDhvkFAb-K3GZeUjjQ@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
	<20110408201807.6dc58ebf@pitrou.net>
	<BANLkTik680zMecEcZCZPJt6BDrLAgOWrpg@mail.gmail.com>
	<20110408210650.24b29d26@pitrou.net>
	<BANLkTi=bqTsGR0+x3v3M1wNzCbG4Tn+a2Q@mail.gmail.com>
	<BANLkTim8Z11Xm5jcKDhvkFAb-K3GZeUjjQ@mail.gmail.com>
Message-ID: <BANLkTi=3xQ9C-z7U3K4LmACEi8y-BV5QDg@mail.gmail.com>

On Fri, Apr 8, 2011 at 3:56 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> Consider how much IO is used in Python and how often people have to deal
> with IO exceptions. ?Why should we have to deal with errnos if there is a
> way that fits into the exception hierarchy? ?It's not that they are obscure.
> ?It's that we don't need the actual numbers or?mnemonics? in our Python
> code. ?We can hide them behind more meaningful exceptions. ?At least, that's
> my take.

You can use more useful exceptions for *some* errnos. But not for all.

-- 
--Guido van Rossum (python.org/~guido)


From python at mrabarnett.plus.com  Sat Apr  9 01:29:45 2011
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 09 Apr 2011 00:29:45 +0100
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTi=nXwgPRoVtfduL0_sMB+tBMH+Swg@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>	<BANLkTimVkUvx+SRJvjT3OADe6HehNpFSew@mail.gmail.com>
	<BANLkTi=nXwgPRoVtfduL0_sMB+tBMH+Swg@mail.gmail.com>
Message-ID: <4D9F9A69.1090305@mrabarnett.plus.com>

On 08/04/2011 18:56, Guido van Rossum wrote:
> On Fri, Apr 8, 2011 at 10:52 AM, Mike Graham<mikegraham at gmail.com>  wrote:
>> On Fri, Apr 8, 2011 at 1:11 PM, Guido van Rossum<guido at python.org>  wrote:
>>> With apologies for not reading the PEP or this thread in full, some comments:
>>>
>>> - I really like the syntax "except<exc>  [as<var>] [if<test>]:".
>>> This addresses a pretty common use case in my experience. I don't care
>>> for the alternate proposed syntax that started this thread. I'm not
>>> sure that the 'if' subclause makes sense without the 'as' subclause,
>>> since most likely you'd want to refer to the caught exception. I note
>>> that it is more powerful than putting "if not<test>: raise" in the
>>> body of the except-clause, because it will allow subsequent except
>>> clauses to match still. I also note that it is a much "cleaner" change
>>> than (again) reorganizing the exception hierarchy, since there is no
>>> backward compatibility to consider.
>>
>> Of course, multiple cases can currently be handled by an if statement
>>
>> except Foo as e:
>>     if e.bar == baz:
>>         ...
>>     elif e.bar == qux:
>>         ...
>>     else:
>>         raise
>>
>> This code isn't so bad: it's not hard to understand, it's an existing
>> pattern, it uses general, powerful constructs.
>
> The problem (as I tried to say, but apparently not clearly enough) is
> that if there's a later more general except clause on the same block
> it won't work. E.g.
>
> except Foo as e:
>    <code as above>
> except Exception:
>    <log traceback>
>
> The 'raise' in your code above does not transfer control to the
> "except Exception:" clause. The current solution is either to have two
> nested try/except blocks, or duplicate the logging code; neither is
> great.
>
[snip]
Unless you have something like "continue except", which means "continue
checking the following 'except' clauses".


From ncoghlan at gmail.com  Sat Apr  9 01:50:04 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 9 Apr 2011 09:50:04 +1000
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
Message-ID: <BANLkTikm8RS+8GiODJ_j56QJ4wV3moiVMQ@mail.gmail.com>

On Sat, Apr 9, 2011 at 12:31 AM, Mike Graham <mikegraham at gmail.com> wrote:
> On Fri, Apr 8, 2011 at 10:13 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On Sat, Apr 9, 2011 at 12:04 AM, Mike Graham <mikegraham at gmail.com> wrote:
>>> Neither of these seem justified for a rare case (this sort of patter
>>> is fairly rare, most notably this one example) when there's nothing
>>> that awful about the current solution.
>>
>> Actually, there is an awful lot of code in the wild that gives
>> incorrect and wildly misleading error messages *because* correctly
>> checking the errno attribute is so rare. PEP 3151 would improve the
>> quality of information provided by a lot of tracebacks and error
>> messages without many of those developers needing to even set finger
>> to keyboard.
>
> But Nick, that's different from what I'm saying is rare.
>
> I'm saying that the situation where we need an if->raise on a constant
> parameter is rare (this being almost the only common case). The issue
> of whether people handle that case being rare is separate. Neither the
> "except Foo as e if f(e):" syntax nor the "except foo(some_errno):"
> pattern propose something that stops people from doing stupid stuff
> like "except IOError" with no check.

Sorry, I misinterpreted "neither of these" as including PEP 3151
itself (which aims to eliminate the need for most errno pattern
matching). I had missed that someone had brought up a second
pattern-matching idea, and it was the two different approaches that
you were referring to.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Sat Apr  9 02:19:25 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 9 Apr 2011 10:19:25 +1000
Subject: [Python-ideas] [Python-Dev] AST Transformation Hooks for Domain
	Specific Languages
In-Reply-To: <1302281450.3336.47.camel@radiator.bos.redhat.com>
References: <BANLkTi=NcHXn0USiQkMMADHy2ZNLgDD3Gw@mail.gmail.com>
	<1302281450.3336.47.camel@radiator.bos.redhat.com>
Message-ID: <BANLkTikjwn53o+Aah6DFcEHev2CxbFTsAg@mail.gmail.com>

(Fixed list to be python-ideas)

On Sat, Apr 9, 2011 at 2:50 AM, David Malcolm <dmalcolm at redhat.com> wrote:
> On Fri, 2011-04-08 at 21:29 +1000, Nick Coghlan wrote:
>> AST Transformation Hooks for Domain Specific Languages
>> ======================================================
>
> This reminds me a lot of Mython:
> ?http://mython.org/
> If you haven't seen it, it's well worth a look.

Ah, very interesting - indeed, compile-time metaprogramming is
definitely what this idea is about (with all of the tremendous power
and potentially major liabilities for readability and debugging that
entails).

> My favourite use case for this kind of thing is having the ability to
> embed shell pipelines into Python code, by transforming bash-style
> syntax into subprocess calls (it's almost possible to do all this in
> regular Python by overloading the | and > operators, but not quite).
>
>> Consider:
>>
>> # In some other module
>> ast.register_dsl("dsl.sql", dsl.sql.TransformAST)
>
> Where is this registered? ? Do you have to import this "other module"
> before importing the module using "dsl.sql" ? ? It sounds like this is
> global state for the interpreter.

Yep - the registration would be with the compiler itself. Technically
a "flat" namespace, but allowing dots in the names means it can mirror
the general Python module namespace.

An alternative would be to use the module namespace rather than a
dedicated one, in which case registration might not be necessary.
However, requiring an explicit registration step to support DSLs
doesn't really bother me, even it means they can't be used as
standalone scripts, but instead must be executed via a module that
registers the DSL. With runpy.run_module and runpy.run_path available,
supporting execution of other scripts isn't a great burden.

>> # In a module using that DSL
>
> How is this usage expressed? ?via the following line?
>
>> import dsl.sql
>
> I see the "import dsl.sql" here, but surely you have to somehow process
> the "import" in order to handle the rest of the parsing.

No, it's expressed by the "from dsl.sql" at the end of the line:

    def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql:

The import is included in the example, simply because that was a
pattern I used in my sample expansion.

As Jon pointed out in his reply, it's basically a very similar idea to
Mython's "quote" block, but leveraging off the existing Function node
to pick up other fun tricks like arguments, decorators and annotations
rather than defining a completely separate statement.

> Where and how would the bytes of the file usage the DSL get converted to
> an in-memory tree representation?
>
> IIRC, manipulating AST nodes in CPython requires some care: the parser
> has its own allocator (PyArena), and the entities it allocates have a
> shared lifetime that ends when PyArena_Free occurs.

They survive if you use PyCF_ONLY_AST (or otherwise get hold of the
AST from Python). The arenas are just Python lists and returning the
AST for use by Python code bumps the reference count of the head node.

>> So there you are, that's the crazy idea. The stoning of the heretic
>> may now commence :)
>
> Or, less violently, take it to python-ideas? ?(though I'm not subscribed
> there, fwiw, make of that what you will)

Yeah, sorry about that - the posting to python-dev was a mistake due
to not properly checking the address auto-complete in Gmail.

> One "exciting" aspect of this is that if someone changes the DSL file,
> the meaning of all of your code changes from under you. ?This may or may
> not be a sane approach to software development :)
> (I also worry what this means e.g. for people writing text editors,
> syntax highlighters, etc; insert usual Alan Perlis quote about syntactic
> sugar causing cancer of the semicolon)

Yeah, there's a way to experiment with this that's much friendlier to
those systems: require that the body of the statement *also* be
standard Python code. Support for non-Python syntax would then be
handled by explicitly embedding it in a string (potentially even the
docstring).

The advantage this would have over existing string parsing techniques
is that the contents would be checked at compile time rather than
runtime, just as Mython does.

> Also, insert usual comments about the need to think about how
> non-CPython implementations of Python would go about implementing such
> ideas.

This is also AST-time, so the back end shouldn't make a big difference.

>> Where this idea came from was the various discussions about "make
>> statement" style constructs and a conversation I had with Eric Snow at
>> Pycon about function definition time really being *too late* to do
>> anything particularly interesting that couldn't already be handled
>> better in other ways. Some tricks Dave Malcolm had done to support
>> Python level manipulation of the AST during compilation also played a
>> big part, as did Eugene Toder's efforts to add an AST optimisation
>> step to the compilation process.
>
> Like I said earlier, have a look at Mython

Indeed, in a lot of ways, this idea is really just an alternative
syntax proposal for Mython "quote" statements, along with the concept
of integrating it into the main compiler.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From eltoder at gmail.com  Sat Apr  9 02:27:50 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Fri, 8 Apr 2011 20:27:50 -0400
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>
	<BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>
Message-ID: <BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>

> Yep, but if you're at the point of using a DSL, that's likely to be
> part of a larger framework which can take care of registering the DSL
> for you before you import any modules that need it.

Not necessarily. And registering makes debugging small pieces and
playing in interactive interpreter less convenient.

> Yeah, the big downside is having to almost completely reimplement the
> AST compiler in order to do it that way (since the built-in one would
> choke on the syntax extension).

I did not realize you're proposing to skip parsing of the body of dsl
function and just tuck all the code into Str literal.
I though the idea was to actually do an AST transformation, i.e. use
Python parser first and rewrite resulting AST into something else.
This is a classical approach and something people already do in
Python, though it's not integrated with the language at the moment
(e.g. there's no quote and no way to get AST for function).
If the idea is to use completely custom syntax in dsl function and
implement custom parsers for dsls, it's harder to do with import hook,
but not too hard. A preprocessor step that puts body of dsl functions
into """ will do.

I don't know if I like the stringification idea. It allows much more
freedom in dsl syntax, but to use that freedom one needs to implement
a parser. Just rewriting AST is simpler. Also, the implementation is
very different from an AST transform -- basically, the grammar needs
to say that after you saw dsl function header the next suite needs to
be saved as a string.

> But what would the eager decorator buy you over just specifying a
> different dialect in the "from" clause?

The point of eager decorator is to avoid the need for registration --
code becomes self-sufficient. It doesn't need "from" clause, I forgot
to delete it. To reiterate, the idea is that the code in form
(straw-man syntax)

@@a.dec(args)
def foo():
   ...

in module b will import module a at compile time and execute a.dec on
foo's AST and resulting AST will be used.
This can be done with a 'from' clause as well, if it gains importing
capabilities -- that's just a syntactic difference. However, this
seems very close to existing decorators, so similar syntax and
semantics seem to make sense.

Eugene


From grosser.meister.morti at gmx.net  Sat Apr  9 06:22:26 2011
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Sat, 09 Apr 2011 06:22:26 +0200
Subject: [Python-ideas] Assignments in list/generator expressions
Message-ID: <4D9FDF02.6080402@gmx.net>

I often have things like this:
 >>> ys = [f(x) for x in xs if f(x)]

Obviously there is a redundant function call. You could rephrase that code to the following in order 
to avoid it:
 >>> ys = [y for y in (f(x) for x in xs) if y]

But I think this is cumbersome and the extra generator overhead is unnecessary.

So I propose this syntax:
 >>> ys = [f(x) as y for x in xs if y]

It could be transformed to this:
 >>> ys = []
 >>> for x in xs:
 >>>    y = f(x)
 >>>    if y:
 >>>       ys.append(y)

It's 6:18am here so I might not think straight and there is something fundamentally wrong with this. 
So please flame away.

	-panzi


From grosser.meister.morti at gmx.net  Sat Apr  9 06:29:44 2011
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Sat, 09 Apr 2011 06:29:44 +0200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4D9FDF02.6080402@gmx.net>
References: <4D9FDF02.6080402@gmx.net>
Message-ID: <4D9FE0B8.4000501@gmx.net>

Before I go to bead another thing I miss in Python. In generator expressions you can unpack elements:
 >>> ys = [f(x1) for x1, x2 in xs]

But sometimes you need not just the unpacked elements but also the original tuple:
 >>> ys = [f(x1, (x1, x2)) for x1, x2 in xs]
or
 >>> ys = [f(x[0], x) for x in xs]

If you need x[0] often in your expression it would be nice if you could write something like this:
 >>> ys = [f(x1, x) for (x1, x2) as x in xs]

So the "as" keyword here would be like the "@" in Haskell. You don't need repetitive __getitem__ 
calls and also don't need to rebuild the tuple. Of course if Python would know that the type of 
whatever is unpacked is a tuple (which is immutalbe) optimizations could create the same bytecode 
from both old variants then from the new one.

Good night,
panzi


From pyideas at rebertia.com  Sat Apr  9 07:23:46 2011
From: pyideas at rebertia.com (Chris Rebert)
Date: Fri, 8 Apr 2011 22:23:46 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4D9FDF02.6080402@gmx.net>
References: <4D9FDF02.6080402@gmx.net>
Message-ID: <BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>

On Fri, Apr 8, 2011 at 9:22 PM, Mathias Panzenb?ck
<grosser.meister.morti at gmx.net> wrote:
>
> I often have things like this:
> >>> ys = [f(x) for x in xs if f(x)]
>
> Obviously there is a redundant function call. You could rephrase that code to the following in order to avoid it:
> >>> ys = [y for y in (f(x) for x in xs) if y]
>
> But I think this is cumbersome and the extra generator overhead is unnecessary.
>
> So I propose this syntax:
> >>> ys = [f(x) as y for x in xs if y]
>
> It could be transformed to this:
> >>> ys = []
> >>> for x in xs:
> >>> ? ?y = f(x)
> >>> ? ?if y:
> >>> ? ? ? ys.append(y)
>
> It's 6:18am here so I might not think straight and there is something fundamentally wrong with this. So please flame away.

This has been proposed previously:
http://mail.python.org/pipermail/python-ideas/2009-June/004946.html
(The syntax variant using "as" comes up part of the way through the thread.)

Cheers,
Chris
--
http://blog.rebertia.com


From cmjohnson.mailinglist at gmail.com  Sat Apr  9 07:35:39 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Fri, 8 Apr 2011 19:35:39 -1000
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>
	<BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>
	<BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>
Message-ID: <BANLkTinnWhQPg_2=O8e9pZGt5-NVRHbb_A@mail.gmail.com>

> I did not realize you're proposing to skip parsing of the body of dsl
> function and just tuck all the code into Str literal.

If I can ask a noobish question, in that case, what's the advantage of
the new syntax over the currently existing ability to have a decorator
parse a docstring? Eg., an SQL DSL could be done today as


import dsl.sql

@dsl.sql
def lookup_address(name : dsl.sql.char, dob : dsl.sql.date):
   """select address
   from people
   where name = {name} and dob = {dob}"""
   pass

If you can't bear to lose the docstring, you could instead write something like,


import dsl.sql

@dsl.sql
def lookup_address(name : dsl.sql.char, dob : dsl.sql.date):
   "Looks up the address of a name and DOB."
   return """select address
   from people
   where name = {name} and dob = {dob}"""



What's the advantage of the "def from" over this? Bear in mind that
this way of doing has the advantage of working with today's Python
syntax highlighters. ;-)


From ericsnowcurrently at gmail.com  Sat Apr  9 08:06:34 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 9 Apr 2011 00:06:34 -0600
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTinnWhQPg_2=O8e9pZGt5-NVRHbb_A@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>
	<BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>
	<BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>
	<BANLkTinnWhQPg_2=O8e9pZGt5-NVRHbb_A@mail.gmail.com>
Message-ID: <BANLkTinGX2G2a5LLnzkqKXEoCfDQikAZWw@mail.gmail.com>

On Fri, Apr 8, 2011 at 11:35 PM, Carl M. Johnson <
cmjohnson.mailinglist at gmail.com> wrote:

> > I did not realize you're proposing to skip parsing of the body of dsl
> > function and just tuck all the code into Str literal.
>
> If I can ask a noobish question, in that case, what's the advantage of
> the new syntax over the currently existing ability to have a decorator
> parse a docstring? Eg., an SQL DSL could be done today as
>
>
> import dsl.sql
>
> @dsl.sql
> def lookup_address(name : dsl.sql.char, dob : dsl.sql.date):
>    """select address
>   from people
>   where name = {name} and dob = {dob}"""
>    pass
>
> If you can't bear to lose the docstring, you could instead write something
> like,
>
>
> import dsl.sql
>
> @dsl.sql
> def lookup_address(name : dsl.sql.char, dob : dsl.sql.date):
>   "Looks up the address of a name and DOB."
>   return """select address
>    from people
>   where name = {name} and dob = {dob}"""
>
>
>
> What's the advantage of the "def from" over this? Bear in mind that
> this way of doing has the advantage of working with today's Python
> syntax highlighters. ;-)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>

My understanding is that the AST transformation is done at compile-time,
while the decorator happens at run-time.  So your .pyc file gets your
transformed code.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110409/b5e06128/attachment.html>

From ncoghlan at gmail.com  Sat Apr  9 11:12:11 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 9 Apr 2011 19:12:11 +1000
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTinGX2G2a5LLnzkqKXEoCfDQikAZWw@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>
	<BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>
	<BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>
	<BANLkTinnWhQPg_2=O8e9pZGt5-NVRHbb_A@mail.gmail.com>
	<BANLkTinGX2G2a5LLnzkqKXEoCfDQikAZWw@mail.gmail.com>
Message-ID: <BANLkTimTKFuZ2Mt1KeaAhoiRWKnwFWszDw@mail.gmail.com>

On Sat, Apr 9, 2011 at 4:06 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> @dsl.sql
>> def lookup_address(name : dsl.sql.char, dob : dsl.sql.date):
>> ? "Looks up the address of a name and DOB."
>> ? return """select address
>> ? from people
>> ? where name = {name} and dob = {dob}"""
>>
>>
>>
>> What's the advantage of the "def from" over this? Bear in mind that
>> this way of doing has the advantage of working with today's Python
>> syntax highlighters. ;-)
>
> My understanding is that the AST transformation is done at compile-time,
> while the decorator happens at run-time. ?So your .pyc file gets your
> transformed code.

Yep - it's the move to compile time metaprogramming that makes ideas
like this one and tools like Mython interesting. Everything else is
too close to what can already be done with decorators and metaclasses
to justify the additional complexity.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Sat Apr  9 11:26:56 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 9 Apr 2011 19:26:56 +1000
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>
	<BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>
	<BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>
Message-ID: <BANLkTimvryikvyvas4Zqh+VjH_5zGJC_Hg@mail.gmail.com>

On Sat, Apr 9, 2011 at 10:27 AM, Eugene Toder <eltoder at gmail.com> wrote:
>> Yep, but if you're at the point of using a DSL, that's likely to be
>> part of a larger framework which can take care of registering the DSL
>> for you before you import any modules that need it.
>
> Not necessarily. And registering makes debugging small pieces and
> playing in interactive interpreter less convenient.

Not really. All you would need is for it to be standard practice for
DSL definition modules to provide a __main__ clause that registers
themselves and uses runpy to execute the passed in arguments (once we
start converting some of the stdlib utilities like "pdb" to do that,
I'll even see if there is sufficiently commonality to make it worth
factoring out a "runpy.delegate_main()" helper function).

So a simple "python -mdsl.sql myfile.py" would run a file that uses
the DSL, while "python -i -mdsl.sql" would get you an interactive
interpreter that understood that DSL dialect.

Embedding imports inside functions has long been fragile, and is a
good recipe for deadlocking code (especially if process forks are
involved). I *really* don't want to advocate enshrining them as an
implicit part of a standard piece of syntax (even if that syntax has
no realistic chance of ever making it into the core language).

>> Yeah, the big downside is having to almost completely reimplement the
>> AST compiler in order to do it that way (since the built-in one would
>> choke on the syntax extension).
>
> I did not realize you're proposing to skip parsing of the body of dsl
> function and just tuck all the code into Str literal.
> I though the idea was to actually do an AST transformation, i.e. use
> Python parser first and rewrite resulting AST into something else.
> This is a classical approach and something people already do in
> Python, though it's not integrated with the language at the moment
> (e.g. there's no quote and no way to get AST for function).

That part is actually orthogonal to the core AST construction hook
concept. It just occurred to me as a possibility when I was writing
the SQL example.

Certainly, if I try implementing this, I won't do it that way - I'll
just add the "syntax" attribute to the Function node and do the
experiment as a true AST transformation process rather than implicitly
quoting the function body. That will make the AST generation tweaks
significantly simpler than the full-blown Mython-style quoting
concept.

> I don't know if I like the stringification idea. It allows much more
> freedom in dsl syntax, but to use that freedom one needs to implement
> a parser. Just rewriting AST is simpler. Also, the implementation is
> very different from an AST transform -- basically, the grammar needs
> to say that after you saw dsl function header the next suite needs to
> be saved as a string.

Reusing Python syntax would still be easy - you'd simply invoke
ast.parse() on the stringified body.

However, I'm not that fond of the idea either, so I doubt I'll
experiment with it (particularly since Mython already works that way).

> in module b will import module a at compile time and execute a.dec on
> foo's AST and resulting AST will be used.
> This can be done with a 'from' clause as well, if it gains importing
> capabilities -- that's just a syntactic difference. However, this
> seems very close to existing decorators, so similar syntax and
> semantics seem to make sense.

It also has the virtue of getting the DSL name in a more prominent
place, and allowing easier composition of DSLs. I'm sold :)

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From eltoder at gmail.com  Sat Apr  9 18:27:21 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Sat, 9 Apr 2011 12:27:21 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
Message-ID: <BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>

Survey of list comprehension features and syntax:

1) Structure
Haskell: [expr | generators], generators = generator (, generator)*
Python: [expr generators], generators = generator+
2) Take
Haskell: x <- xs
Python: for x in xs
3) Filter
Haskell: cond
Python: if cond
4) Local definition
Haskell: let name = expr
Python: strangely missing

I'd rather add syntax equivalent to Haskell's let, than inventing
completely new syntax. E.g. with name = expr. Original from the first
post becomes

ys = [y for x in xs with y = f(x) if y]

but it's more flexible in general.

Eugene


From eltoder at gmail.com  Sat Apr  9 19:00:42 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Sat, 9 Apr 2011 13:00:42 -0400
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTimvryikvyvas4Zqh+VjH_5zGJC_Hg@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>
	<BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>
	<BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>
	<BANLkTimvryikvyvas4Zqh+VjH_5zGJC_Hg@mail.gmail.com>
Message-ID: <BANLkTimDkoMkwaUNgTeD8QsdZ=aYcakLqg@mail.gmail.com>

>> My understanding is that the AST transformation is done at compile-time,
>> while the decorator happens at run-time.  So your .pyc file gets your
>> transformed code.
>
>Yep - it's the move to compile time metaprogramming that makes ideas
>like this one and tools like Mython interesting.
In python compile time and run time are less separated than in many
other languages. I don't see a big difference between doing
transformation at compile time proper vs. at import time (except for
start-up time, if the transformation is taking seconds). Technically,
decorator that parses docstring is as powerful as an AST
transformation. The difference is mostly about end user friendliness
and aesthetics -- whether you feel like you're using a feature or a
cheap hack.

> So a simple "python -mdsl.sql myfile.py" would run a file that uses
> the DSL, while "python -i -mdsl.sql" would get you an interactive
> interpreter that understood that DSL dialect.
Ok, it's easy to do, but to me the fact that I have to even think
about it makes it less convenient.

> Reusing Python syntax would still be easy - you'd simply invoke
> ast.parse() on the stringified body.
Right, otherwise I would knew that I *don't* like the idea. At the
moment I don't know if I like it :)
Using Python parser first leads to a slippery slope: Python syntax is
only tuned for one language -- Python. If we start using it as a base
for DSLs more widely, we'll want some generic extensions, which would
give syntax error in python, but would produce AST nodes for DSLs to
transform. E.g. space delimited expression list ('select foo') or
custom suite ('join:\nleft\nright\ncond').
Stringifying function body avoids this problem (though one would have
to write a full parser even if we wants a very little tweak to python
syntax), but it allows completely non-python syntax mixed with python.
I don't know if it's a big problem, though.

Eugene


From python at mrabarnett.plus.com  Sat Apr  9 19:55:48 2011
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 09 Apr 2011 18:55:48 +0100
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
Message-ID: <4DA09DA4.4070306@mrabarnett.plus.com>

On 09/04/2011 17:27, Eugene Toder wrote:
> Survey of list comprehension features and syntax:
>
> 1) Structure
> Haskell: [expr | generators], generators = generator (, generator)*
> Python: [expr generators], generators = generator+
> 2) Take
> Haskell: x<- xs
> Python: for x in xs
> 3) Filter
> Haskell: cond
> Python: if cond
> 4) Local definition
> Haskell: let name = expr
> Python: strangely missing
>
> I'd rather add syntax equivalent to Haskell's let, than inventing
> completely new syntax. E.g. with name = expr. Original from the first
> post becomes
>
> ys = [y for x in xs with y = f(x) if y]
>
> but it's more flexible in general.
>
Or possibly:

     ys = [y for x in xs with f(x) as y if y]


From g.brandl at gmx.net  Sat Apr  9 20:55:34 2011
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 09 Apr 2011 20:55:34 +0200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
Message-ID: <inqa36$ln1$2@dough.gmane.org>

On 09.04.2011 18:27, Eugene Toder wrote:
> Survey of list comprehension features and syntax:
> 
> 1) Structure
> Haskell: [expr | generators], generators = generator (, generator)*
> Python: [expr generators], generators = generator+
> 2) Take
> Haskell: x <- xs
> Python: for x in xs
> 3) Filter
> Haskell: cond
> Python: if cond
> 4) Local definition
> Haskell: let name = expr
> Python: strangely missing

Uh, that's hardly strange considering that Python doesn't have local
assignments in expressions anywhere.

Georg



From eltoder at gmail.com  Sat Apr  9 21:04:07 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Sat, 9 Apr 2011 15:04:07 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <inqa36$ln1$2@dough.gmane.org>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<inqa36$ln1$2@dough.gmane.org>
Message-ID: <BANLkTikwmU9eJGkShE0_ymBCLKwbnJEqtA@mail.gmail.com>

> Uh, that's hardly strange considering that Python doesn't have local
> assignments in expressions anywhere.

Well, depending on your definition of "assignments in expression" you
can say that Haskell doesn't have them either.
Or you can say that Python has them since list comprehension was
introduced -- for x in xs is an assignment to x. In fact, I'm reminded
that local assignment in list comprehension is easily emulated with
for:

ys = [y for x in xs for y in [f(x)] if y]

Eugene


From tjreedy at udel.edu  Sat Apr  9 23:10:18 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 09 Apr 2011 17:10:18 -0400
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTinGX2G2a5LLnzkqKXEoCfDQikAZWw@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>	<BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>	<BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>	<BANLkTinnWhQPg_2=O8e9pZGt5-NVRHbb_A@mail.gmail.com>
	<BANLkTinGX2G2a5LLnzkqKXEoCfDQikAZWw@mail.gmail.com>
Message-ID: <inqhvo$rod$1@dough.gmane.org>

On 4/9/2011 2:06 AM, Eric Snow wrote:

> My understanding is that the AST transformation is done at compile-time,
> while the decorator happens at run-time.

Nope. Decorator call happens just after compilation, but before calls.
A decorator could compile a new code object and replace the original.

If the decorator return a wrapper instead of the original function, 
then, of course, the *wrapper* gets called with each call. Or a 
decorator can return a replacement that does not call the original function.

-- 
Terry Jan Reedy



From arnodel at gmail.com  Sat Apr  9 23:37:55 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Sat, 9 Apr 2011 22:37:55 +0100
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <inqhvo$rod$1@dough.gmane.org>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>	<BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>	<BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>	<BANLkTinnWhQPg_2=O8e9pZGt5-NVRHbb_A@mail.gmail.com>
	<BANLkTinGX2G2a5LLnzkqKXEoCfDQikAZWw@mail.gmail.com>
	<inqhvo$rod$1@dough.gmane.org>
Message-ID: <93EDEA7F-EB1E-42E4-BA9D-45E32AA7D689@gmail.com>


On 9 Apr 2011, at 22:10, Terry Reedy wrote:

> On 4/9/2011 2:06 AM, Eric Snow wrote:
> 
>> My understanding is that the AST transformation is done at compile-time,
>> while the decorator happens at run-time.
> 
> Nope. Decorator call happens just after compilation, but before calls.
> A decorator could compile a new code object and replace the original.

I don't understand what you are saying.  A decorator call occurs just after the execution of the def statement it decorates, which is definitely at run-time.  And that's completely unrelated to compilation time.  In fact most module are executed many times between compilations.  And yes, a decorator can compile a new code object, but how is it relevant?

-- 
Arnaud



From tjreedy at udel.edu  Sun Apr 10 00:13:13 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 09 Apr 2011 18:13:13 -0400
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <93EDEA7F-EB1E-42E4-BA9D-45E32AA7D689@gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>	<BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>	<BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>	<BANLkTinnWhQPg_2=O8e9pZGt5-NVRHbb_A@mail.gmail.com>	<BANLkTinGX2G2a5LLnzkqKXEoCfDQikAZWw@mail.gmail.com>	<inqhvo$rod$1@dough.gmane.org>
	<93EDEA7F-EB1E-42E4-BA9D-45E32AA7D689@gmail.com>
Message-ID: <inqllm$cfg$1@dough.gmane.org>

On 4/9/2011 5:37 PM, Arnaud Delobelle wrote:
>
> On 9 Apr 2011, at 22:10, Terry Reedy wrote:
>
>> On 4/9/2011 2:06 AM, Eric Snow wrote:
>>
>>> My understanding is that the AST transformation is done at
>>> compile-time, while the decorator happens at run-time.
>>
>> Nope. Decorator call happens just after compilation, but before
>> calls. A decorator could compile a new code object and replace the
>> original.
>
> I don't understand what you are saying.

I was thinking Eric meant function call time, but that is probably 
wrong, so ignore my comment.

-- 
Terry Jan Reedy



From cmjohnson.mailinglist at gmail.com  Sun Apr 10 01:44:12 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Sat, 9 Apr 2011 13:44:12 -1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikwmU9eJGkShE0_ymBCLKwbnJEqtA@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<inqa36$ln1$2@dough.gmane.org>
	<BANLkTikwmU9eJGkShE0_ymBCLKwbnJEqtA@mail.gmail.com>
Message-ID: <BANLkTik_0WzDAT8DZBQWbTxvMd+73HskQA@mail.gmail.com>

The lack of local assignment is a feature. It keeps people from trying
to write overly complex one-liners. It is better to separate each step
out into a generator expression on a different line of the final
program, so that things don't become so dense you can't tell at a
glance what's going on. Or heck, you could just use a for-loop if what
you're trying to do is complicated enough.


From debatem1 at gmail.com  Sun Apr 10 02:17:48 2011
From: debatem1 at gmail.com (geremy condra)
Date: Sat, 9 Apr 2011 17:17:48 -0700
Subject: [Python-ideas] Developer wish list
Message-ID: <BANLkTinj_hAxDP7U7cJUEr7jzc7cj-urag@mail.gmail.com>

Earlier this year there was some discussion[0] about putting up a page
on the wiki where developers could list the feature proposals they
most wanted and most hated for the benefit of those posting to
python-ideas. It's taken me a while to get around to it, but I've put
up a skeleton for the page at [1] and would love it if some of you
guys would take a look, let me know what you like/don't like, and
maybe even post a few of your pet projects or pet peeves.

Thanks for your time and effort,
Geremy Condra

[0]: http://mail.python.org/pipermail/python-ideas/2011-March/009230.html
[1]: http://wiki.python.org/moin/wishlist


From bruce at leapyear.org  Sun Apr 10 02:42:48 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Sat, 9 Apr 2011 17:42:48 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikwmU9eJGkShE0_ymBCLKwbnJEqtA@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<inqa36$ln1$2@dough.gmane.org>
	<BANLkTikwmU9eJGkShE0_ymBCLKwbnJEqtA@mail.gmail.com>
Message-ID: <BANLkTinYye20DVCwKMKgMJs+KCFC15JY2Q@mail.gmail.com>

On Fri, Apr 8, 2011 at 9:22 PM, Mathias Panzenb?ck <
grosser.meister.morti at gmx.net> wrote:

> So I propose this syntax:
> >>> ys = *[f(x) as y for x in xs if y]*


On Sat, Apr 9, 2011 at 10:55 AM, MRAB <python at mrabarnett.plus.com> wrote:

> Or possibly:
>    ys =* [y for x in xs with f(x) as y if y]*
>

I find both of these hard to read, just like a sentence with multiple
non-parallel dependent phrases. The meaning of y as x is reversed from x in
y. As noted, there are two ways to do this that work already. I find the
first one quite readable:

On Fri, Apr 8, 2011 at 9:22 PM, Mathias Panzenb?ck <
grosser.meister.morti at gmx.net> wrote:

> Obviously there is a redundant function call. You could rephrase that code
> to the following in order to avoid it:
> >>> ys = *[y for y in (f(x) for x in xs) if y]*
>

On Sat, Apr 9, 2011 at 12:04 PM, Eugene Toder <eltoder at gmail.com> wrote:

> local assignment in list comprehension is easily emulated with for:

ys = *[y for x in xs for y in [f(x)] if y]*


I wonder if the "solution" to this is to publish one of these as the answer
to the common question of how to do this. Timing tests indicate that the
first version is faster.

--- Bruce
*New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April
Fools!
*New!** *Blog post:
http://www.vroospeak.com/2011/04/march-gets-more-madness-next-year.html April
Fools!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110409/964783ab/attachment.html>

From ncoghlan at gmail.com  Sun Apr 10 09:27:58 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 10 Apr 2011 17:27:58 +1000
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <BANLkTimDkoMkwaUNgTeD8QsdZ=aYcakLqg@mail.gmail.com>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>
	<BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>
	<BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>
	<BANLkTimvryikvyvas4Zqh+VjH_5zGJC_Hg@mail.gmail.com>
	<BANLkTimDkoMkwaUNgTeD8QsdZ=aYcakLqg@mail.gmail.com>
Message-ID: <BANLkTi=1k3t-Lvm6Lbk-6+4QFuY6T1hoEg@mail.gmail.com>

On Sun, Apr 10, 2011 at 3:00 AM, Eugene Toder <eltoder at gmail.com> wrote:
>>Yep - it's the move to compile time metaprogramming that makes ideas
>>like this one and tools like Mython interesting.
> In python compile time and run time are less separated than in many
> other languages. I don't see a big difference between doing
> transformation at compile time proper vs. at import time (except for
> start-up time, if the transformation is taking seconds). Technically,
> decorator that parses docstring is as powerful as an AST
> transformation. The difference is mostly about end user friendliness
> and aesthetics -- whether you feel like you're using a feature or a
> cheap hack.

The difference is actually pretty huge. If it is done at compile time:

1. You consume the entire module at once, permitting non-local
effects. This is significant, as it allows you to reference variables
in outer scopes, converting them into cell references (e.g. see the
"all_nonlocal" example from my original post). Runtime is far too late
to do that, since the compiler has already finished the symbol table
analysis and code generation that handles nested scopes.

2. Compile time operations can have their results cached in the
generated PYC files. This cannot happen with runtime operations.

If this was handled as a runtime operation, you couldn't really do
anything that can't already be done with decorators and metaclasses.

People sometimes get confused about how Python's compilation and
execution model differs from that of a static language like C. To give
a quick rundown of the different major phases:

C build-and-execution model:
  - compile time (.c -> .o)
  - link time (multiple .o -> executable)
  - dynamic linking (loading additional modules at runtime)
  - runtime (actual code execution)

Only the last 2 when the program is executed

Python
  - compile time (.py -> bytecode)
  - definition time (only significant for functions and classes - time
when the "def" or "class" statement is executed)
  - runtime (actual code execution, guaranteed to be after definition
time for code inside a function body and during definition time for a
class body)

Since compilation is implicit, and there is no pre-linking step, all
of these steps happen when the program is executed (although the first
step can optionally be performed in advance).

It's the separation of compile and definition time that is the major
difference between a scripting language like Python and a more
traditional language like C. In a traditional language, the
"top-level" code is handled entirely by the compiler, and never
actually touched at runtime, so you can't do things like use loops or
conditional logic or exception handling to affect how your program is
defined (and if you can, the syntax will typically be completely
different from the "normal" syntax of the language). In a scripting
language, top-level code has access to all the same constructs as code
inside functions (and, typically, vice-versa - hence first class
functions and type definitions).

Currently Python lets you do lots of things at runtime (i.e. most
code) and at definition time (decorators, metaclasses, default
arguments, annotations). There are, however, no compile time hooks
other than creating your own import hook as Mython does, and
completely taking over the compilation process.

>> So a simple "python -mdsl.sql myfile.py" would run a file that uses
>> the DSL, while "python -i -mdsl.sql" would get you an interactive
>> interpreter that understood that DSL dialect.
> Ok, it's easy to do, but to me the fact that I have to even think
> about it makes it less convenient.

Yes, but it's the only way to make this work as a compile-time
operation (since compilation is completed before module execution
starts). If it's runtime only, then there's no point in doing it at
all. Decorators and metaclasses have that space well and truly
covered.

>> Reusing Python syntax would still be easy - you'd simply invoke
>> ast.parse() on the stringified body.
> Right, otherwise I would knew that I *don't* like the idea. At the
> moment I don't know if I like it :)
> Using Python parser first leads to a slippery slope: Python syntax is
> only tuned for one language -- Python. If we start using it as a base
> for DSLs more widely, we'll want some generic extensions, which would
> give syntax error in python, but would produce AST nodes for DSLs to
> transform. E.g. space delimited expression list ('select foo') or
> custom suite ('join:\nleft\nright\ncond').
> Stringifying function body avoids this problem (though one would have
> to write a full parser even if we wants a very little tweak to python
> syntax), but it allows completely non-python syntax mixed with python.
> I don't know if it's a big problem, though.

Python-AST is a reasonable place to start though, since non-Python
syntax can easily be written inside a docstring. It also creates a
subtle social pressure in favour of staying within the spirit of
Python syntax and semantics, and clearly demarcating (via
triple-quoted strings) when you're straying away from that.

An SQL DSL, for example, would most likely go the route of
triple-quoting the entire SQL statement, but could also do something
more novel like using assignments to define SQL clauses:

    select = address
    tables = people
    where = name == {name} and dob == {dob}

Such is the power and danger of compile-time metaprogramming :)

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Sun Apr 10 09:37:40 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 10 Apr 2011 17:37:40 +1000
Subject: [Python-ideas] AST Transformation Hooks for Domain Specific
	Languages
In-Reply-To: <inqllm$cfg$1@dough.gmane.org>
References: <BANLkTimKgVmY_UrjyCJxdqbaf+Ks=gZs2Q@mail.gmail.com>
	<BANLkTin9-PL1zBjvykzBAky0DLMygSkYxg@mail.gmail.com>
	<BANLkTin8wpA6YL4rOjV4pjYvX1tSbWyhCw@mail.gmail.com>
	<BANLkTikFd3BoV3LXVhF3SEfHQH=guyryUg@mail.gmail.com>
	<BANLkTimio4jBheEadzn7oWxYNCqbmyHhOQ@mail.gmail.com>
	<BANLkTinnWhQPg_2=O8e9pZGt5-NVRHbb_A@mail.gmail.com>
	<BANLkTinGX2G2a5LLnzkqKXEoCfDQikAZWw@mail.gmail.com>
	<inqhvo$rod$1@dough.gmane.org>
	<93EDEA7F-EB1E-42E4-BA9D-45E32AA7D689@gmail.com>
	<inqllm$cfg$1@dough.gmane.org>
Message-ID: <BANLkTinNvw2E8mR7D1=_ChjoEzOOjJ6kaQ@mail.gmail.com>

On Sun, Apr 10, 2011 at 8:13 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> I was thinking Eric meant function call time, but that is probably wrong, so
> ignore my comment.

Yeah, he was talking about function definition time rather than call
time. From the point-of-view of PYC file generation, both function
definition time and function call time are "runtime" events that
happen after the PYC file has been generated.

The typical flow of events is...

- module compilation (including PYC caching)
- module execution (including definition of top-level functions and classes)
- function execution (including definition of any nested functions and classes)

The latter two are both "runtime", but functions are special because
key events happens at all three stages in the chain:

- At compilation time, the compiler symbol table analysis and code
generation figures out all the variable scoping and decides whether to
emit local, nonlocal or global operations for variable access, and
which variables need to be stored in cells for correct access from
closures
- At function definition time, default arguments, annotations and
decorators are all evaluated, any cell references are linked up to the
relevant outer scopes and the function name is bound in the containing
scope
- At function call time, the arguments are populated and the actual
function body is executed

Generators add a fourth stage into the process, since they separate
generator construction time from iteration time (and don't check their
arguments until they start iterating).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Sun Apr 10 09:41:49 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 10 Apr 2011 17:41:49 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
Message-ID: <BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>

On Sun, Apr 10, 2011 at 2:27 AM, Eugene Toder <eltoder at gmail.com> wrote:
> I'd rather add syntax equivalent to Haskell's let, than inventing
> completely new syntax. E.g. with name = expr. Original from the first
> post becomes
>
> ys = [y for x in xs with y = f(x) if y]
>
> but it's more flexible in general.

It isn't strangely missing at all, it's just hard:
http://www.python.org/dev/peps/pep-3150/

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From arnodel at gmail.com  Sun Apr 10 12:37:42 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Sun, 10 Apr 2011 11:37:42 +0100
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
Message-ID: <9FD3A096-1857-4B55-AAB1-BCF6A0C16FB7@gmail.com>


On 10 Apr 2011, at 08:41, Nick Coghlan wrote:

> On Sun, Apr 10, 2011 at 2:27 AM, Eugene Toder <eltoder at gmail.com> wrote:
>> I'd rather add syntax equivalent to Haskell's let, than inventing
>> completely new syntax. E.g. with name = expr. Original from the first
>> post becomes
>> 
>> ys = [y for x in xs with y = f(x) if y]
>> 
>> but it's more flexible in general.
> 
> It isn't strangely missing at all, it's just hard:
> http://www.python.org/dev/peps/pep-3150/


I have just read the proposal and I would like to suggest an idea for implementing it.  It may not be a good idea but at least it's simple!  Given the "torture test", at the end of the proposal:

b = {}
a = b[f(a)] = x given:
    x = 42
    def f(x):
        return x
assert "x" not in locals()
assert "f" not in locals()
assert a == 42
assert d[42] == 42 given:
    d = b
assert "d" not in locals()b = {}

Without "given", one would write it as below.  Obviously it succeeds at module, local and class scope. 

b = {}
x = 42
def f(x):
    return x
a = b[f(a)] = x
del x
del f
assert "x" not in locals()
assert "f" not in locals()
assert a == 42
d = b
assert d[42] == 42
del d
assert "d" not in locals()

The only (but big!) problem is that this deletes any previously defined variable "x", "f", and "d" in the current scope.  So why not just rename the variables "x", "f", and "d" at compile time to something that is guaranteed not to appear elsewhere in the scope, e.g. "@1", "@2", "@3" or some other naming scheme, in the style of the old "gensym" function in Lisp?

So we get:

b = {}
@1 = 42
def @2(x):
    return x
a = b[@2(a)] = @1
del @1
del @2
assert "x" not in locals()
assert "f" not in locals()
assert a == 42
@3 = b
assert @3[42] == 42
del @3
assert "d" not in locals()

-- 
Arnaud



From ncoghlan at gmail.com  Sun Apr 10 14:10:00 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 10 Apr 2011 22:10:00 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <9FD3A096-1857-4B55-AAB1-BCF6A0C16FB7@gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<9FD3A096-1857-4B55-AAB1-BCF6A0C16FB7@gmail.com>
Message-ID: <BANLkTimtnSGe7DOD6+Dv7LzSBohh08kKvQ@mail.gmail.com>

On Sun, Apr 10, 2011 at 8:37 PM, Arnaud Delobelle <arnodel at gmail.com> wrote:
> The only (but big!) problem is that this deletes any previously defined variable "x", "f", and "d" in the current scope. ?So why not just rename the variables "x", "f", and "d" at compile time to something that is guaranteed not to appear elsewhere in the scope, e.g. "@1", "@2", "@3" or some other naming scheme, in the style of the old "gensym" function in Lisp?

I considered a strategy along those lines when eliminating the leakage
of the scope variables for list comprehensions in Py3k and it turns
out to be rather painful from a nested scopes point of view. It's not
*impossible*, of course, but you end up having to manage this parallel
pseudo-namespace inside the real one and it rapidly becomes an
absolute mess (as you have to implement a second copy of most of the
nested scopes logic in order to handle closures correctly and you will
curse the existence of the locals() function).

The compiler can't see inside the strings used to look up values via
"locals()", so there's no way to adjust them to handle the automatic
renaming. If you "lift" a class or function out of a statement local
namespace, there's also the question of what the value of the __name__
attribute ends up being.

I have updated the torture test accordingly, it now includes a third
example that renaming strategies will have serious problems handling:

   y = y given:
       x = 42
       def f(): pass
       y = locals("x"), f.__name__
   assert "x" not in locals()
   assert "f" not in locals()
   assert y == (42, "f")

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From eltoder at gmail.com  Sun Apr 10 15:31:53 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Sun, 10 Apr 2011 09:31:53 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
Message-ID: <BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>

> It isn't strangely missing at all, it's just hard:
> http://www.python.org/dev/peps/pep-3150/

Local definition in list comprehension is significantly simpler that
'given'. Semantically, 'with name = expr' can be treated as a more
readable form of 'for name in [expr]'. Implementation can use a simple
assignment.

Eugene


From lac at openend.se  Sun Apr 10 17:13:18 2011
From: lac at openend.se (Laura Creighton)
Date: Sun, 10 Apr 2011 17:13:18 +0200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: Message from Eugene Toder <eltoder@gmail.com> of "Sun,
	10 Apr 2011 09:31:53 EDT."
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com> 
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com> 
Message-ID: <201104101513.p3AFDI94025509@theraft.openend.se>

In a message of Sun, 10 Apr 2011 09:31:53 EDT, Eugene Toder writes:
>> It isn't strangely missing at all, it's just hard:
>> http://www.python.org/dev/peps/pep-3150/
>
>Local definition in list comprehension is significantly simpler that
>'given'. Semantically, 'with name = expr' can be treated as a more
>readable form of 'for name in [expr]'. Implementation can use a simple
>assignment.
>
>Eugene

We'll have to disagree then.  I think 'for name in [expr]' is a whole
lot more readable than 'with name = expr'.  Indeed, my recent
experience in code reading had shown, I thought, that a good number of
people are using the with statement whenever they possibly can, to the
detriment of readability.

Comments like this make me wonder if the problem is that those people
actually find those constructs more readable.  If we largely cannot
agree on what makes for more readable code, then I think we have a
different problem than the one I thought we had (people showing off
their knowledge of how to use every new feature).

Laura



From ncoghlan at gmail.com  Sun Apr 10 17:18:25 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Apr 2011 01:18:25 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
Message-ID: <BANLkTin5G0yPrAAY7R_FDckBTyrowABZ=w@mail.gmail.com>

On Sun, Apr 10, 2011 at 11:31 PM, Eugene Toder <eltoder at gmail.com> wrote:
>> It isn't strangely missing at all, it's just hard:
>> http://www.python.org/dev/peps/pep-3150/
>
> Local definition in list comprehension is significantly simpler that
> 'given'. Semantically, 'with name = expr' can be treated as a more
> readable form of 'for name in [expr]'. Implementation can use a simple
> assignment.

Part of the point of PEP 3150 is that such requests don't *stay*
simple. If one subexpression, why not two? If in a list comprehension,
why not in a conditional expression? The "you can't do arbitrary
embedded assignments in expressions" rule is at least a simple one,
even if some folks don't like it (most likely due to a
mathematics/functional programming background, where the idea of
persistent state isn't a basic assumption the way it is in normal
imperative programming).

While naming the iteration variables is a fundamental part of writing
comprehensions, naming other subexpressions is not, so it doesn't make
sense to provide comprehension specific syntax for it. If you want to
name subexpressions, the "one obvious way" is to use an explicit loop,
typically as part of a generator:

def transform(xs):
  """Meaningful description of whatever the transform does"""
  for x in xs:
    y = f(x)
    if y:
      yield y

ys = transform(xs)

The problem of "I want to use this value as part of a conditional
expression and in its own right" actually arises in more locations
than just filtering comprehensions - it also comes up for it
statements, while loops and conditional expressions. In all cases, you
run up against the limits of what expressions allow and have to devote
a bunch of vertical whitespace to switch to using expressions instead.

The issue you're really up against is the fact that Guido made a
choice 20 years ago to enforce a statement/expression dichotomy and to
keep name binding entirely the purview of statements. Comprehension
iteration variables are an exception that were added later on, but
they're used in a very constrained way that matches the embedded
assignment of a specific statement type (i.e. the header line on for
loops).

Embedded assignment proposals for conditions suffer badly from the
fact that there is no statement level equivalent for the kind of
assignment they propose. The obvious solution (allowing the
conditional to be named) is too limiting, but solutions that are
sufficiently flexible to be worthwhile end up allowing naming of
arbitrary subexpressions, effectively throwing away a fundamental
design principle of the language. And so the discussion dies, until
the next time someone decides that their personal use case for
embedded assignment is worth altering the language to allow.

Mathias's proposal in this case is a slight improvement over past
suggestions, since it allows the conditional expression to differ from
the value used in the rest of the expression. However, I'm not sure it
would be feasible to implement it in a way that constrained it to
comprehensions only. Even if that was possible, fixing only 1 out of
the 4 parts of the language where the problem arises isn't much of a
solution from a broader point of view.

And it still only works in simple cases, being utterly unhelpful for cases like:

  {f(k):g(v) for k, v in d.items() if f(k) and g(v)}

As past discussions have shown, the only thing really powerful enough
to cover an acceptably large number of use cases is full-blown
embedded assignment for arbitrary expressions:

ys = [y for x in xs if not (f(x) as y)]
{k2:v2 for k, v in d.items() if (f(k) as k2) and (g(v) as v2)}

That's an awfully big sledgehammer for a rather small nail :P

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Sun Apr 10 17:29:52 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Apr 2011 01:29:52 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <201104101513.p3AFDI94025509@theraft.openend.se>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
Message-ID: <BANLkTiko3kLERL=JpLHG+Gqhb0FEuMbYKQ@mail.gmail.com>

On Mon, Apr 11, 2011 at 1:13 AM, Laura Creighton <lac at openend.se> wrote:
> Comments like this make me wonder if the problem is that those people
> actually find those constructs more readable. ?If we largely cannot
> agree on what makes for more readable code, then I think we have a
> different problem than the one I thought we had (people showing off
> their knowledge of how to use every new feature).

I don't think it's an either/or problem - I think both of those things
can cause problems with code readability.

One of the first questions I want to ask people that try to push
Python's comprehension syntax to (and past) its limits is whether they
have come to Python programming from a mathematics or pure functional
programming background. If they have, then "normal" imperative
programming will feel understandably unnatural to them, despite it
being the idiomatic approach to Python programming (with
comprehensions only being employed for those relatively simple cases
they were really designed to handle).

But I also see people using metaclasses for things that should be
decorators, context managers for things that should be ordinary higher
order functions, higher order functions for things that should just be
plain boring old functions, closures for things that should be
classes... all of those I tend to chalk up to "when you have just
discovered a nice shiny new hammer, everything looks like a nail (even
when you still have the saw, screwdriver and pliers in the toolkit)".

Or, as I once heard it put, some code overuses powerful constructs to
the point where it is like "using a bazooka to kill a fly" :)

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From anikom15 at gmail.com  Sun Apr 10 19:13:37 2011
From: anikom15 at gmail.com (Westley =?ISO-8859-1?Q?Mart=EDnez?=)
Date: Sun, 10 Apr 2011 10:13:37 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTin5G0yPrAAY7R_FDckBTyrowABZ=w@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<BANLkTin5G0yPrAAY7R_FDckBTyrowABZ=w@mail.gmail.com>
Message-ID: <1302455617.13880.8.camel@localhost.localdomain>

On Mon, 2011-04-11 at 01:18 +1000, Nick Coghlan wrote:
> On Sun, Apr 10, 2011 at 11:31 PM, Eugene Toder <eltoder at gmail.com> wrote:
> >> It isn't strangely missing at all, it's just hard:
> >> http://www.python.org/dev/peps/pep-3150/
> >
> > Local definition in list comprehension is significantly simpler that
> > 'given'. Semantically, 'with name = expr' can be treated as a more
> > readable form of 'for name in [expr]'. Implementation can use a simple
> > assignment.
> 
> Part of the point of PEP 3150 is that such requests don't *stay*
> simple. If one subexpression, why not two? If in a list comprehension,
> why not in a conditional expression? The "you can't do arbitrary
> embedded assignments in expressions" rule is at least a simple one,
> even if some folks don't like it (most likely due to a
> mathematics/functional programming background, where the idea of
> persistent state isn't a basic assumption the way it is in normal
> imperative programming).
> 
> While naming the iteration variables is a fundamental part of writing
> comprehensions, naming other subexpressions is not, so it doesn't make
> sense to provide comprehension specific syntax for it. If you want to
> name subexpressions, the "one obvious way" is to use an explicit loop,
> typically as part of a generator:
> 
> def transform(xs):
>   """Meaningful description of whatever the transform does"""
>   for x in xs:
>     y = f(x)
>     if y:
>       yield y
> 
> ys = transform(xs)
> 
> The problem of "I want to use this value as part of a conditional
> expression and in its own right" actually arises in more locations
> than just filtering comprehensions - it also comes up for it
> statements, while loops and conditional expressions. In all cases, you
> run up against the limits of what expressions allow and have to devote
> a bunch of vertical whitespace to switch to using expressions instead.
> 
> The issue you're really up against is the fact that Guido made a
> choice 20 years ago to enforce a statement/expression dichotomy and to
> keep name binding entirely the purview of statements. Comprehension
> iteration variables are an exception that were added later on, but
> they're used in a very constrained way that matches the embedded
> assignment of a specific statement type (i.e. the header line on for
> loops).
> 
> Embedded assignment proposals for conditions suffer badly from the
> fact that there is no statement level equivalent for the kind of
> assignment they propose. The obvious solution (allowing the
> conditional to be named) is too limiting, but solutions that are
> sufficiently flexible to be worthwhile end up allowing naming of
> arbitrary subexpressions, effectively throwing away a fundamental
> design principle of the language. And so the discussion dies, until
> the next time someone decides that their personal use case for
> embedded assignment is worth altering the language to allow.
> 
> Mathias's proposal in this case is a slight improvement over past
> suggestions, since it allows the conditional expression to differ from
> the value used in the rest of the expression. However, I'm not sure it
> would be feasible to implement it in a way that constrained it to
> comprehensions only. Even if that was possible, fixing only 1 out of
> the 4 parts of the language where the problem arises isn't much of a
> solution from a broader point of view.
> 
> And it still only works in simple cases, being utterly unhelpful for cases like:
> 
>   {f(k):g(v) for k, v in d.items() if f(k) and g(v)}
> 
> As past discussions have shown, the only thing really powerful enough
> to cover an acceptably large number of use cases is full-blown
> embedded assignment for arbitrary expressions:
> 
> ys = [y for x in xs if not (f(x) as y)]
> {k2:v2 for k, v in d.items() if (f(k) as k2) and (g(v) as v2)}
> 
> That's an awfully big sledgehammer for a rather small nail :P
> 
> Cheers,
> Nick.
> 

I don't quite understand the obsession of having an entire function on
one line.  In programming, one line is usually associated with one
action.  Python is all about readability, and it's probably best to put
this kind of stuff in a function on multiple lines, not only will it be
easier to read, it'll be reusable and easier to modify.



From guido at python.org  Sun Apr 10 19:42:57 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 10 Apr 2011 10:42:57 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
Message-ID: <BANLkTimn=BO+fr1GPDTKHDaXvtVciZrhcQ@mail.gmail.com>

On Sun, Apr 10, 2011 at 6:31 AM, Eugene Toder <eltoder at gmail.com> wrote:
>> It isn't strangely missing at all, it's just hard:
>> http://www.python.org/dev/peps/pep-3150/
>
> Local definition in list comprehension is significantly simpler that
> 'given'. Semantically, 'with name = expr' can be treated as a more
> readable form of 'for name in [expr]'. Implementation can use a simple
> assignment.

Eww. Since we already have "with expr as name" I don't think we should
add "with name = expr", even if it's in a different context.

OTOH I am personally a big fan of PEP 3150; now that the moratorium is
lifted I hope that someone will attempt a proper implementation of it
so we can find out all the odd corner cases (which I'm sure will be
plentiful given that this is quite a novel thing for Python).

BTW also +1 on Laura's mini-rant on readability. A lot of folks have
ideas that would add a minor shortcut to the language, without making
a big difference in true expressivity, but with a big potential cost
in readability. (Remember Perl, the ultimate write-only language.) In
defense of 'given' I think that it adds something truly new (variables
that go out of scope before the function containing them returns) in a
very readable way. And the "definition follows use" thing has worked
out pretty well in list comprehensions and generator expressions, if
you ask me.

-- 
--Guido van Rossum (python.org/~guido)


From eltoder at gmail.com  Sun Apr 10 20:31:38 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Sun, 10 Apr 2011 14:31:38 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <201104101513.p3AFDI94025509@theraft.openend.se>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
Message-ID: <BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>

> We'll have to disagree then. ?I think 'for name in [expr]' is a whole
> lot more readable than 'with name = expr'. ?Indeed, my recent
> experience in code reading had shown, I thought, that a good number of
> people are using the with statement

It's well known that minor syntax details cause a lot of disagreement
(http://www.haskell.org/haskellwiki/Wadlers_Law), so that's not very
surprising. However, are you sure you're not confusing python's with
statement and 'with' word in my email, which is simply a placeholder
for appropriate keyword to mirror Haskell's 'let'? Or was this just an
occasion to complain about with statement abuse, unrelated to list
comprehension discussion?

If I was not clear: I used 'with' keyword for demonstration purposes
only, to show what the syntax *might* look like. I do not suggest to
reuse 'with' (even though python has a history keywords reuse, e.g.
'else'), it's just a placeholder.

> While naming the iteration variables is a fundamental part of writing
> comprehensions, naming other subexpressions is not, so it doesn't make
> sense to provide comprehension specific syntax for it.

Well, apparently designers of Haskell, Scala, F# and Clojure decided
that it does make sense to provide a comprehension specific syntax for
naming subexpressions in comprehensions.

Eugene


From arnodel at gmail.com  Sun Apr 10 21:28:41 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Sun, 10 Apr 2011 20:28:41 +0100
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTimtnSGe7DOD6+Dv7LzSBohh08kKvQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<9FD3A096-1857-4B55-AAB1-BCF6A0C16FB7@gmail.com>
	<BANLkTimtnSGe7DOD6+Dv7LzSBohh08kKvQ@mail.gmail.com>
Message-ID: <4DB86F2D-5741-48C8-99D2-362FC62E1B62@gmail.com>


On 10 Apr 2011, at 13:10, Nick Coghlan wrote:

> On Sun, Apr 10, 2011 at 8:37 PM, Arnaud Delobelle <arnodel at gmail.com> wrote:
>> The only (but big!) problem is that this deletes any previously defined variable "x", "f", and "d" in the current scope.  So why not just rename the variables "x", "f", and "d" at compile time to something that is guaranteed not to appear elsewhere in the scope, e.g. "@1", "@2", "@3" or some other naming scheme, in the style of the old "gensym" function in Lisp?
> 
> I considered a strategy along those lines when eliminating the leakage
> of the scope variables for list comprehensions in Py3k and it turns
> out to be rather painful from a nested scopes point of view. It's not
> *impossible*, of course, but you end up having to manage this parallel
> pseudo-namespace inside the real one and it rapidly becomes an
> absolute mess (as you have to implement a second copy of most of the
> nested scopes logic in order to handle closures correctly and you will
> curse the existence of the locals() function).
> 
> The compiler can't see inside the strings used to look up values via
> "locals()", so there's no way to adjust them to handle the automatic
> renaming. If you "lift" a class or function out of a statement local
> namespace, there's also the question of what the value of the __name__
> attribute ends up being.
> 
> I have updated the torture test accordingly, it now includes a third
> example that renaming strategies will have serious problems handling:
> 
>   y = y given:
>       x = 42
>       def f(): pass
>       y = locals("x"), f.__name__
>   assert "x" not in locals()
>   assert "f" not in locals()
>   assert y == (42, "f")

OK.  It's clear I don't know enough about the compiling process to be able to understand all the problems.  The f.__name__ issue doesn't seem insurmountable, but I can't see how to get around the locals()["x"] issue in a sane manner (I see what you mean about cursing the locals() function!).  BTW, I think there is a typo in the example above: shouldn't

    y = locals("x"), ...

be

    y = locals()["x"], ...

-- 
Arnaud



From greg.ewing at canterbury.ac.nz  Sun Apr 10 23:09:50 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 11 Apr 2011 09:09:50 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <201104101513.p3AFDI94025509@theraft.openend.se>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
Message-ID: <4DA21C9E.4080602@canterbury.ac.nz>

Laura Creighton wrote:

> We'll have to disagree then.  I think 'for name in [expr]' is a whole
> lot more readable than 'with name = expr'.

I think I would find 'where name = expr' slightly more
readable and aesthetically pleasing than either
'with name = expr' or 'for name in [expr]'.

However if 'for name in [expr]' were optimised to use
an assignment instead of a for-loop, I might consider
using it more often. Currently I would be reluctant to
do so because it feels so inefficient.

-- 
Greg


From guido at python.org  Mon Apr 11 00:48:11 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 10 Apr 2011 15:48:11 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4DB86F2D-5741-48C8-99D2-362FC62E1B62@gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<9FD3A096-1857-4B55-AAB1-BCF6A0C16FB7@gmail.com>
	<BANLkTimtnSGe7DOD6+Dv7LzSBohh08kKvQ@mail.gmail.com>
	<4DB86F2D-5741-48C8-99D2-362FC62E1B62@gmail.com>
Message-ID: <BANLkTinKG001aXnmO=BW0Kc1Xa=jF=Xk_w@mail.gmail.com>

On Sun, Apr 10, 2011 at 12:28 PM, Arnaud Delobelle <arnodel at gmail.com> wrote:
>
> On 10 Apr 2011, at 13:10, Nick Coghlan wrote:
>
>> On Sun, Apr 10, 2011 at 8:37 PM, Arnaud Delobelle <arnodel at gmail.com> wrote:
>>> The only (but big!) problem is that this deletes any previously defined variable "x", "f", and "d" in the current scope. ?So why not just rename the variables "x", "f", and "d" at compile time to something that is guaranteed not to appear elsewhere in the scope, e.g. "@1", "@2", "@3" or some other naming scheme, in the style of the old "gensym" function in Lisp?
>>
>> I considered a strategy along those lines when eliminating the leakage
>> of the scope variables for list comprehensions in Py3k and it turns
>> out to be rather painful from a nested scopes point of view. It's not
>> *impossible*, of course, but you end up having to manage this parallel
>> pseudo-namespace inside the real one and it rapidly becomes an
>> absolute mess (as you have to implement a second copy of most of the
>> nested scopes logic in order to handle closures correctly and you will
>> curse the existence of the locals() function).
>>
>> The compiler can't see inside the strings used to look up values via
>> "locals()", so there's no way to adjust them to handle the automatic
>> renaming. If you "lift" a class or function out of a statement local
>> namespace, there's also the question of what the value of the __name__
>> attribute ends up being.
>>
>> I have updated the torture test accordingly, it now includes a third
>> example that renaming strategies will have serious problems handling:
>>
>> ? y = y given:
>> ? ? ? x = 42
>> ? ? ? def f(): pass
>> ? ? ? y = locals("x"), f.__name__
>> ? assert "x" not in locals()
>> ? assert "f" not in locals()
>> ? assert y == (42, "f")
>
> OK. ?It's clear I don't know enough about the compiling process to be able to understand all the problems. ?The f.__name__ issue doesn't seem insurmountable, but I can't see how to get around the locals()["x"] issue in a sane manner (I see what you mean about cursing the locals() function!). ?BTW, I think there is a typo in the example above: shouldn't
>
> ? ?y = locals("x"), ...
>
> be
>
> ? ?y = locals()["x"], ...

FWIW, I don't mind if an implementation of this (or anything new,
really) doesn't work well with locals().

-- 
--Guido van Rossum (python.org/~guido)


From guido at python.org  Mon Apr 11 01:06:01 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 10 Apr 2011 16:06:01 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
Message-ID: <BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>

On Sun, Apr 10, 2011 at 11:31 AM, Eugene Toder <eltoder at gmail.com> wrote:
>> We'll have to disagree then. ?I think 'for name in [expr]' is a whole
>> lot more readable than 'with name = expr'. ?Indeed, my recent
>> experience in code reading had shown, I thought, that a good number of
>> people are using the with statement

Eugene, I'm pretty sure you didn't write that, but as I'm not
following this megathread in real time I don't know who did. Can you
please be so kind to leave attributions in? (I'm glad you snip stuff
you don't need, but the attributions are essential.)

> If I was not clear: I used 'with' keyword for demonstration purposes
> only, to show what the syntax *might* look like. I do not suggest to
> reuse 'with' (even though python has a history keywords reuse, e.g.
> 'else'), it's just a placeholder.

It was not clear.

>> While naming the iteration variables is a fundamental part of writing
>> comprehensions, naming other subexpressions is not, so it doesn't make
>> sense to provide comprehension specific syntax for it.
>
> Well, apparently designers of Haskell, Scala, F# and Clojure decided
> that it does make sense to provide a comprehension specific syntax for
> naming subexpressions in comprehensions.

If the designers of Haskell, Scala, F# and Clojure jumped off a cliff,
would you jump too? :-)

Those languages are all considerably more "functional" in nature than
Python will ever be.

It's hard to express Python's design principles exactly, but
"following other languages" has never been high on the list. We do
borrow good ideas, but tend to reinterpret them in the context of
Python's existing semantics, syntax and constraints. We also place a
high value on growing the language very slowly and carefully; too many
shiny new features are more likely to distract than to help, even if
each one individually is a great idea. And we have a history of stuff
we do and don't like, which may not always be rational, but which has
served us well up to here. If you don't like it, you can always write
one of those other languages...

(That's not to say that I am vetoing a specific proposal. But the bar
is high, and inventing a "Pythonic" syntax is a prerequisite. So is a
good story with a set of examples explaining why the new feature adds
something you cannot easily do today.)

-- 
--Guido van Rossum (python.org/~guido)


From lac at openend.se  Mon Apr 11 01:08:08 2011
From: lac at openend.se (Laura Creighton)
Date: Mon, 11 Apr 2011 01:08:08 +0200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: Message from Eugene Toder <eltoder@gmail.com> of "Sun,
	10 Apr 2011 14:31:38 EDT."
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com> 
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com> 
Message-ID: <201104102308.p3AN88DD003570@theraft.openend.se>

In a message of Sun, 10 Apr 2011 14:31:38 EDT, Eugene Toder writes:
<snip>
>It's well known that minor syntax details cause a lot of disagreement
>(http://www.haskell.org/haskellwiki/Wadlers_Law), so that's not very
>surprising. However, are you sure you're not confusing python's with
>statement and 'with' word in my email, which is simply a placeholder
>for appropriate keyword to mirror Haskell's 'let'? Or was this just an
>occasion to complain about with statement abuse, unrelated to list
>comprehension discussion?
<snip>
>Eugene

I actually had something else in mind.  One problem that you get whenever
you extend a language 'i.e. make it more expressive' is that it now becomes
possible for people to write old things they want to do in new ways.  And,
of course, this is what you want for the case of <whatever you want the
more expressiveness for>, but it is difficult to may a syntax change that
doesn't admit a whole lot of other possibilities as well.

The argument I have heard around here is 'good programmers don't write
unreadable code, as a matter of personal virtue'.  This ignores the
fact that I have to read a lot of code written by either not-so-good
or not-so-virtuous programmers, but I am coming to the conclusion that
the problem may be deeper than that.  I now believe that some of the
hard-to-read stuff I am reading is written by people who actually find
this way of writing easier to read and understand than the sort I
prefer.

Thus I now think that 'expressiveness' is a limited human good, not an
absolute one, (like health).  You should want you language to be in the
golden middle between 'not epressive enough' and 'too expressive'.  If
your language is sufficiently limited, then idioms will show up, and
become widely adopted precisely because they are one of only a few ways
(ideally the only way) to do it.  This already pleases the people who 
find this way to do things readable, and those who would naturally pick
a different way of doing things are stuck with it, and possibly over time
and with familiarity will come to find this way readable as well. 

But with too much expressibility there comes a danger we will fragment into
different sub-communities, those who think that syntax X is more readable
than syntax Y and vice versa.  There is no reason to believe that these
linguistic communities will ever converge. This is the hidden cost of allowing 
more expressiveness in your language.

Laura




From eltoder at gmail.com  Mon Apr 11 02:18:35 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Sun, 10 Apr 2011 20:18:35 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
Message-ID: <BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>

On Sun, Apr 10, 2011 at 7:06 PM, Guido van Rossum <guido at python.org> wrote:
> Can you please be so kind to leave attributions in? (I'm glad you snip stuff
> you don't need, but the attributions are essential.)
>
<snip>
>
> It was not clear.

Will try to improve on both accounts.

On Sun, Apr 10, 2011 at 7:06 PM, Guido van Rossum <guido at python.org> wrote:
> (That's not to say that I am vetoing a specific proposal. But the bar
> is high, and inventing a "Pythonic" syntax is a prerequisite. So is a
> good story with a set of examples explaining why the new feature adds
> something you cannot easily do today.)

This particular feature is not the top on my wish list either. I just
don't see a good explanation for why it's missing. It's a lesser-known
feature in Haskell (no mention in Wikipedia! :-), but it is handy at
times. As mentioned above, people suggest it for Python from time to
time.

As for the syntax, there are many choices. Going with <keyword> name =
expr, <keyword> can be 'let', as in Haskell, F# and Clojure, or
'where' as Greg suggested above, or 'given' as it's related to PEP
3150 -- the scope of name partially extends to the left of it's
definition. From the backward compatibility point, 'let' is likely too
short for a new keyword, and PEP 3150 describes the problem with
'where'.

ys = [y for x in xs where y = f(x) if y]

ys = [y for x in xs given y = f(x) if y]

There are likely other words that will do. Alternatively, some
punctuation can be used, e.g. semicolon:

ys = [y for x in xs; y = f(x) if y]

Eugene


From lac at openend.se  Mon Apr 11 02:39:36 2011
From: lac at openend.se (Laura Creighton)
Date: Mon, 11 Apr 2011 02:39:36 +0200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: Message from Eugene Toder <eltoder@gmail.com> of "Sun,
	10 Apr 2011 20:18:35 EDT."
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com> 
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com> 
Message-ID: <201104110039.p3B0daqp011821@theraft.openend.se>

In a message of Sun, 10 Apr 2011 20:18:35 EDT, Eugene Toder writes:
<snip>
>This particular feature is not the top on my wish list either. I just
>don't see a good explanation for why it's missing. It's a lesser-known
>feature in Haskell (no mention in Wikipedia! :-), but it is handy at
>times. As mentioned above, people suggest it for Python from time to
>time.
<snip>

The problem, as I see it is that you see it as ``missing''.  You have
some internal model of computer languages and not having it disagrees
with your notion of 'completeness' or 'symmetry' or 'consistency' or
something along those lines.

But the reason I was drawn to python in the first place was that it was
an elegant language, which in practice meant that it didn't have a lot
of cruft I found in other languages.  After all, if I had wanted to use
those other languages, I know where to find them.  And I actually enjoy
using Haskell, on those very rare occasions when I have a problem that
I find suited to the language.  

But turning Python more Haskell-like has no appeal to me.  I need a set
of very concrete use cases of 'I am trying to do this' and either 'I cannot'
or 'I am doing it this way and oh, how slow/ugly/error prone/ it is!'
before I am interested in a proposal.  Change for the sake of completeness
has no value to me.

I may be blessed in that I live in Gothenburg, Sweden, where Chalmers University
is one of the great centres of Haskell-admiration.  So I have many friends
who love the language so much they try to do everything in it.  Debugging a
failing webserver written in Haskell has been painful.  There is real
value in using whitespace and indentation to separate logical ideas, and
jamming everything into a comprehension completely destroys this.  The
aesthetic gain in symmetry comes a very, very, distant second to me in
this race.

Laura



From jsbueno at python.org.br  Mon Apr 11 02:56:07 2011
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Sun, 10 Apr 2011 21:56:07 -0300
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4D9FDF02.6080402@gmx.net>
References: <4D9FDF02.6080402@gmx.net>
Message-ID: <BANLkTikYVsb3HeAv_isVC-HFG6vmtafvkg@mail.gmail.com>

On Sat, Apr 9, 2011 at 1:22 AM, Mathias Panzenb?ck
<grosser.meister.morti at gmx.net> wrote:
> I often have things like this:
>>>> ys = [f(x) for x in xs if f(x)]
>
> Obviously there is a redundant function call. You could rephrase that code
> to the following in order to avoid it:
>>>> ys = [y for y in (f(x) for x in xs) if y]
>
> But I think this is cumbersome and the extra generator overhead is
> unnecessary.
>
> So I propose this syntax:
>>>> ys = [f(x) as y for x in xs if y]
>
> It could be transformed to this:
>>>> ys = []
>>>> for x in xs:
>>>> ? ?y = f(x)
>>>> ? ?if y:
>>>> ? ? ? ys.append(y)
>
> It's 6:18am here so I might not think straight and there is something
> fundamentally wrong with this. So please flame away.
>

>>> ys = [f(x) as y for x in xs if y]
I supose the above example could be expressed like this:

ys = filter(None, (f(x) for x in xs))

(or

ys = list( filter(None, (f(x) for x in xs)))

if you happen to really need a list
)
In nearly any python version. And in doing so, I join the chorus of
-1 for such assignment possibilities.

And I am one that  likes doing big things in one liners from time to time,
just for the fun of it - while in-generator assignment would turn
these one liners much more flexible, it would, as several others
have pointed, imply in the possibility of having rather unreadable code
inside programs.

Regards,

  js
 -><-


> ? ? ? ?-panzi


From stephen at xemacs.org  Mon Apr 11 03:48:14 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 11 Apr 2011 10:48:14 +0900
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <201104102308.p3AN88DD003570@theraft.openend.se>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<201104102308.p3AN88DD003570@theraft.openend.se>
Message-ID: <87y63hblsx.fsf@uwakimon.sk.tsukuba.ac.jp>

Laura Creighton writes:

 > I now believe that some of the hard-to-read stuff I am reading is
 > written by people who actually find this way of writing easier to
 > read and understand than the sort I prefer.

It's not clear to me that they find it easier to understand, only
easier to write.  That is, I think it would be interesting to collect
a sample of each kind and see which is buggier. :-)

Unfortunately, it would be hard work to do that well.


From eltoder at gmail.com  Mon Apr 11 03:42:46 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Sun, 10 Apr 2011 21:42:46 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <201104110039.p3B0daqp011821@theraft.openend.se>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
Message-ID: <BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>

On Sun, Apr 10, 2011 at 8:39 PM, Laura Creighton <lac at openend.se> wrote:
> The problem, as I see it is that you see it as ``missing''. ?You have
> some internal model of computer languages and not having it disagrees
> with your notion of 'completeness' or 'symmetry' or 'consistency' or
> something along those lines.

I just like to have a reasonable explanation for everything I see. A
good story is what often separates a feature from a bug.

On Sun, Apr 10, 2011 at 8:39 PM, Laura Creighton <lac at openend.se> wrote:
> There is real value in using whitespace and indentation to separate logical
> ideas, and jamming everything into a comprehension completely destroys this.

I don't think it's fair to judge a feature by trying to imagine the
worst misuse one can do with it. If you would do this consistently,
you'd never introduce arrays (think code using arrays instead of
classes, with magic undocumented indexes instead of attribute names)
or dynamically-typed scripting languages (think Perl).
Also, as pointed above, local assignment doesn't allow any more code
written as comprehension than one can write now -- there are multiple
work-arounds: repetition, nested comprehensions, for name in [expr].
Local assignment simply provides a more direct way of doing it, and to
me a more direct way is easier to read.

Eugene


From ncoghlan at gmail.com  Mon Apr 11 04:33:50 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Apr 2011 12:33:50 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTinKG001aXnmO=BW0Kc1Xa=jF=Xk_w@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<9FD3A096-1857-4B55-AAB1-BCF6A0C16FB7@gmail.com>
	<BANLkTimtnSGe7DOD6+Dv7LzSBohh08kKvQ@mail.gmail.com>
	<4DB86F2D-5741-48C8-99D2-362FC62E1B62@gmail.com>
	<BANLkTinKG001aXnmO=BW0Kc1Xa=jF=Xk_w@mail.gmail.com>
Message-ID: <BANLkTikeYUVhqe4Sq08h+d6Z8G70FDWgBw@mail.gmail.com>

On Mon, Apr 11, 2011 at 8:48 AM, Guido van Rossum <guido at python.org> wrote:
> On Sun, Apr 10, 2011 at 12:28 PM, Arnaud Delobelle <arnodel at gmail.com> wrote:
>> OK. ?It's clear I don't know enough about the compiling process to be able to understand all the problems. ?The f.__name__ issue doesn't seem insurmountable, but I can't see how to get around the locals()["x"] issue in a sane manner (I see what you mean about cursing the locals() function!). ?BTW, I think there is a typo in the example above: shouldn't
>>
>> ? ?y = locals("x"), ...
>>
>> be
>>
>> ? ?y = locals()["x"], ...
>
> FWIW, I don't mind if an implementation of this (or anything new,
> really) doesn't work well with locals().

While supporting locals() correctly is a bit of a pain, I find "Does
read access to locals() still work?" to be a useful proxy for "Will
proposing this feature have authors of Python debugging tools massing
outside my front door with pitchforks and torches?" :)

I don't think it's a trade-off that has to be made in the specific
case of PEP 3150 - the copy-in/copy-out idea should be viable and
reasonably intuitive given Python's existing scoping rules.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From bruce at leapyear.org  Mon Apr 11 04:51:24 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Sun, 10 Apr 2011 19:51:24 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
Message-ID: <BANLkTi=HAQSsZQKBm0GzT-TfR07edmtp9g@mail.gmail.com>

On Sun, Apr 10, 2011 at 6:42 PM, Eugene Toder <eltoder at gmail.com> wrote:

> Local assignment simply provides a more direct way of doing it, and to
> me a more direct way is easier to read.


I don't think there's going to be consensus that sticking additional
assignment in the middle of a generator expression makes it more readable.
The expressions you're talking about really are conceptually nested and
you're trying to eliminate/hide that. Let me throw out an alternative. In
these expressions there's redundant boilerplate ("x for x in ...") when what
we're really doing is a simple filter or map across the sequence. In these
examples, {sequence} stands in for both the sequence being iterated and the
value from the sequence being operated on.

( for {seq} + 1 )   =>  ( x + 1 for x in seq )
( for f({seq}) )    =>  ( f(x) for x in seq )
( if {seq} > 1 }    =>  ( x for x in seq if x > 1 )

But here's the catch. While I would like a simpler way to express these
idioms, I don't think these offer enough of an advantage. And I think that
when you start to nest them it quickly becomes unreadable:


(y for y in (f(x) for x in xs) if y)  =>  (if {(for f({xs})})


And {} looks  a bit too much like () for my taste -- I find that trailing
})}) particularly hard to read.

So why offer this proposal? I think there's a natural tendency to try to
come up with terse expressions for common idioms. But they don't add power
and they make the language less approachable. I think terseness is sometimes
the opposite of readability. In C# the lambda syntax is terse and I find it
unreadable if written without line breaks:

f(x, y + 1, x => x + 1, y + 1)

vs.

f(x, y + 1,
  x => x + 1,
  y + 1)


--- Bruce
*New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April
Fools!
*New!** *Blog post:
http://www.vroospeak.com/2011/04/march-gets-more-madness-next-year.html April
Fools!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110410/9f7f8944/attachment.html>

From ncoghlan at gmail.com  Mon Apr 11 04:58:00 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Apr 2011 12:58:00 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
Message-ID: <BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>

On Mon, Apr 11, 2011 at 11:42 AM, Eugene Toder <eltoder at gmail.com> wrote:
> I don't think it's fair to judge a feature by trying to imagine the
> worst misuse one can do with it. If you would do this consistently,
> you'd never introduce arrays (think code using arrays instead of
> classes, with magic undocumented indexes instead of attribute names)
> or dynamically-typed scripting languages (think Perl).
> Also, as pointed above, local assignment doesn't allow any more code
> written as comprehension than one can write now -- there are multiple
> work-arounds: repetition, nested comprehensions, for name in [expr].
> Local assignment simply provides a more direct way of doing it, and to
> me a more direct way is easier to read.

And we already have one kind of expression that owes its existence
almost entirely to the suboptimal workaround people were using to get
around the fact it wasn't available* :)

A "given/as" list comprehension subclause inspired by PEP 3150 might
actually be a more interesting idea to explore than I first thought
(considering that PEP 3150 itself is absolutely no help at all for
this particular use case).

That is, instead of having to choose one of the following 4
alternatives (and assorted variants of the explicit loop, such as
invoking list() on a generator function):

ys = [f(x) for x in xs if f(x)]
ys = [y for y in (f(x) for x in xs) if y]
ys = [y for x in xs for y in [f(x)] if y]

def _build_sequence(xs):
    ys = []
    for x in xs:
        y = f(x):
        if y:
            ys.append(y)
    return ys
ys = _build_sequence(xs)

The language could offer an "obvious" answer of the form:

ys = [y for x in xs given f(x) as y if y]

Multiple subexpressions would be handled gracefully via tuple
unpacking, and the translation to "long-form" Python code for actual
execution by the eval loop would follow the same style as is already
used for comprehensions and generator expressions (i.e. the last
alternative I listed above, with the details of the emitted code
changing appropriately based on the kind of comprehension).

So, despite my initial negative reaction, I'd now be interested in
reading a fully fleshed out PEP for this proposal.

Cheers,
Nick.

(*Read PEP 308 if the reference is unfamiliar)

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Mon Apr 11 05:00:09 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Apr 2011 13:00:09 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
Message-ID: <BANLkTikriD8EhCyyNGxv4FjBYYp3yP_eZw@mail.gmail.com>

On Mon, Apr 11, 2011 at 12:58 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> def _build_sequence(xs):
> ? ?ys = []
> ? ?for x in xs:
> ? ? ? ?y = f(x):
> ? ? ? ?if y:
> ? ? ? ? ? ?ys.append(y)
> ? ?return ys
> ys = _build_sequence(xs)

Oops, random colon slipped in as typo. That should be:

def _build_sequence(xs):
   ys = []
   for x in xs:
       y = f(x)
       if y:
           ys.append(y)
   return ys
ys = _build_sequence(xs)

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Mon Apr 11 05:03:51 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Apr 2011 13:03:51 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <201104102308.p3AN88DD003570@theraft.openend.se>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<eltoder@gmail.com>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<201104102308.p3AN88DD003570@theraft.openend.se>
Message-ID: <BANLkTikhBZvROhf6OfBhaR3suuhdJGDMkg@mail.gmail.com>

On Mon, Apr 11, 2011 at 9:08 AM, Laura Creighton <lac at openend.se> wrote:
> But with too much expressibility there comes a danger we will fragment into
> different sub-communities, those who think that syntax X is more readable
> than syntax Y and vice versa. ?There is no reason to believe that these
> linguistic communities will ever converge. This is the hidden cost of allowing
> more expressiveness in your language.

This is certainly one of points that comes to my mind when folks are
speculating as to reasons why Lisp never became more popular than it
did. (There are plenty of other reason put forward of course, I just
think this is one of the often overlooked possibilities).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From guido at python.org  Mon Apr 11 05:03:44 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 10 Apr 2011 20:03:44 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
Message-ID: <BANLkTi=zAsVyk2Z9bh5FP1S=PpbgyVunWQ@mail.gmail.com>

On Sun, Apr 10, 2011 at 6:42 PM, Eugene Toder <eltoder at gmail.com> wrote:
> On Sun, Apr 10, 2011 at 8:39 PM, Laura Creighton <lac at openend.se> wrote:
>> The problem, as I see it is that you see it as ``missing''. ?You have
>> some internal model of computer languages and not having it disagrees
>> with your notion of 'completeness' or 'symmetry' or 'consistency' or
>> something along those lines.
>
> I just like to have a reasonable explanation for everything I see. A
> good story is what often separates a feature from a bug.

Your continued insistence on hearing an explanation why it is missing
is beginning to sound more and more like asking when Python stopped
beating his wife.

(This is not unusual. We get a lot of questions phrased as "why does
Python not do X?" Do you think that is a good way to get a discussion
started?)

-- 
--Guido van Rossum (python.org/~guido)


From guido at python.org  Mon Apr 11 05:10:27 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 10 Apr 2011 20:10:27 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikeYUVhqe4Sq08h+d6Z8G70FDWgBw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<9FD3A096-1857-4B55-AAB1-BCF6A0C16FB7@gmail.com>
	<BANLkTimtnSGe7DOD6+Dv7LzSBohh08kKvQ@mail.gmail.com>
	<4DB86F2D-5741-48C8-99D2-362FC62E1B62@gmail.com>
	<BANLkTinKG001aXnmO=BW0Kc1Xa=jF=Xk_w@mail.gmail.com>
	<BANLkTikeYUVhqe4Sq08h+d6Z8G70FDWgBw@mail.gmail.com>
Message-ID: <BANLkTinD9F3CKUKhVKzn-UkzQDW3jXwdVw@mail.gmail.com>

On Sun, Apr 10, 2011 at 7:33 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Mon, Apr 11, 2011 at 8:48 AM, Guido van Rossum <guido at python.org> wrote:
>> FWIW, I don't mind if an implementation of this (or anything new,
>> really) doesn't work well with locals().
>
> While supporting locals() correctly is a bit of a pain, I find "Does
> read access to locals() still work?" to be a useful proxy for "Will
> proposing this feature have authors of Python debugging tools massing
> outside my front door with pitchforks and torches?" :)

Maybe, but debugger may be given access to things that regular code
does not, and if it has to work a little harder to get at the values
of local variables that is okay in my book. So I'd rather see the
actual question asked ("how can a debugger make sense of this") than a
proxy for that question. (And sure, if it's easy to make it work with
locals(), I am not objecting. :-)

-- 
--Guido van Rossum (python.org/~guido)


From eltoder at gmail.com  Mon Apr 11 06:10:38 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Mon, 11 Apr 2011 00:10:38 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
Message-ID: <BANLkTinMC+fiyuWb8p-NEiiQVciKT=yttw@mail.gmail.com>

On Sun, Apr 10, 2011 at 10:58 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> And we already have one kind of expression that owes its existence
> almost entirely to the suboptimal workaround people were using to get
> around the fact it wasn't available* :)

And 5 years later there are still people out there who prefer using
and/or trick. They tend to say something about the new ways being too
new and the old ways being good enough :)

On Sun, Apr 10, 2011 at 11:03 PM, Guido van Rossum <guido at python.org> wrote:
> Your continued insistence on hearing an explanation why it is missing
> is beginning to sound more and more like asking when Python stopped
> beating his wife.

Sorry, I don't get the reference.

On Sun, Apr 10, 2011 at 11:03 PM, Guido van Rossum <guido at python.org> wrote:
> (This is not unusual. We get a lot of questions phrased as "why does
> Python not do X?" Do you think that is a good way to get a discussion
> started?)

In my defense, I'm actually more interested in the history than in
adding this feature, as I find the lack of it a pretty small niggle.
The discussion was started with specific examples and suggestions, but
it was not very interesting so far, because many (not all) responses
are a generic 'new features are bad because they are new' and it's
very hard to argue with that :-) And I do not dispute the fact that
new features need a high level of scrutiny.

Eugene


From anikom15 at gmail.com  Mon Apr 11 06:10:56 2011
From: anikom15 at gmail.com (Westley =?ISO-8859-1?Q?Mart=EDnez?=)
Date: Sun, 10 Apr 2011 21:10:56 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
Message-ID: <1302495056.18762.4.camel@localhost.localdomain>

On Sun, 2011-04-10 at 20:18 -0400, Eugene Toder wrote:
> On Sun, Apr 10, 2011 at 7:06 PM, Guido van Rossum <guido at python.org> wrote:
> > Can you please be so kind to leave attributions in? (I'm glad you snip stuff
> > you don't need, but the attributions are essential.)
> >
> <snip>
> >
> > It was not clear.
> 
> Will try to improve on both accounts.
> 
> On Sun, Apr 10, 2011 at 7:06 PM, Guido van Rossum <guido at python.org> wrote:
> > (That's not to say that I am vetoing a specific proposal. But the bar
> > is high, and inventing a "Pythonic" syntax is a prerequisite. So is a
> > good story with a set of examples explaining why the new feature adds
> > something you cannot easily do today.)
> 
> This particular feature is not the top on my wish list either. I just
> don't see a good explanation for why it's missing. It's a lesser-known
> feature in Haskell (no mention in Wikipedia! :-), but it is handy at
> times. As mentioned above, people suggest it for Python from time to
> time.
> 
> As for the syntax, there are many choices. Going with <keyword> name =
> expr, <keyword> can be 'let', as in Haskell, F# and Clojure, or
> 'where' as Greg suggested above, or 'given' as it's related to PEP
> 3150 -- the scope of name partially extends to the left of it's
> definition. From the backward compatibility point, 'let' is likely too
> short for a new keyword, and PEP 3150 describes the problem with
> 'where'.
> 
> ys = [y for x in xs where y = f(x) if y]
> 
> ys = [y for x in xs given y = f(x) if y]
> 
> There are likely other words that will do. Alternatively, some
> punctuation can be used, e.g. semicolon:
> 
> ys = [y for x in xs; y = f(x) if y]
> 
> Eugene

Semi-colon is already in use and to me (perhaps because of C) tells me
they are unrelated statements. where to me seems the more Pythonic.



From eltoder at gmail.com  Mon Apr 11 06:16:08 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Mon, 11 Apr 2011 00:16:08 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <1302495056.18762.4.camel@localhost.localdomain>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<1302495056.18762.4.camel@localhost.localdomain>
Message-ID: <BANLkTin5Fdm3havXWM-XLBws+Zpscw0yxQ@mail.gmail.com>

On Mon, Apr 11, 2011 at 12:10 AM, Westley Mart?nez <anikom15 at gmail.com> wrote:
> Semi-colon is already in use and to me (perhaps because of C) tells me
> they are unrelated statements. where to me seems the more Pythonic.

Right, keyword is definitely more Pythonic. The problem with 'where'
is that it's already used as identifier by various SQL frameworks,
which is why PEP 3150 uses 'given' instead.

Eugene


From ncoghlan at gmail.com  Mon Apr 11 06:20:56 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Apr 2011 14:20:56 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTimn=BO+fr1GPDTKHDaXvtVciZrhcQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<BANLkTimn=BO+fr1GPDTKHDaXvtVciZrhcQ@mail.gmail.com>
Message-ID: <BANLkTikAv8tzo19kbVHsxOw6ggk--foVTQ@mail.gmail.com>

On Mon, Apr 11, 2011 at 3:42 AM, Guido van Rossum <guido at python.org> wrote:
> OTOH I am personally a big fan of PEP 3150; now that the moratorium is
> lifted I hope that someone will attempt a proper implementation of it
> so we can find out all the odd corner cases (which I'm sure will be
> plentiful given that this is quite a novel thing for Python).

Not *quite* as novel as one might first think - the specific proposal
I currently have in there has its roots in the way list/set/dict
comprehensions work under the hood (in CPython, anyway). So we know
the core principle is sound, although I'm sure you're correct that
there are more corner cases still to be found in generalising the
comprehension-style copy-in/copy-out semantics appropriately.

I also thought of a somewhat decent use case for the feature, too:
"builder" code in module and class namespaces, where you need some
temporary functions, variables and loops to create a named variable,
but then have to make sure to clean up your temporary storage
locations to avoid polluting the public namespace. In the past, I'd
mostly been thinking in terms of function namespaces, and keeping
those "clean" is nowhere near as important as the situation for
classes and modules (it's quite likely someone else has brought this
up in the past, .

To lift some examples from
https://bitbucket.org/pypy/pypy/src/default/lib_pypy/disassembler.py
(which is what got me thinking along these lines)

from opcode import __all__ as _opcodes_all
__all__ = ["dis","disassemble","distb","disco"] + _opcodes_all
del _opcodes_all

Could become:

__all__ = ["dis","disassemble","distb","disco"] + opcode.__all__ given:
    import opcode

And:

def _setup():
    for opcode in opname:
        if not opcode.startswith('<'):
            class O(Opcode):
                pass
            opcode = opcode.replace('+', '_')
            O.__name__ = opcode
            globals()[opcode] = O

_setup()

Could become:

pass given: # I'll add "pass" to the list of supported statements in PEP 3150
    for opcode in opname:
        if not opcode.startswith('<'):
            class O(Opcode):
                pass
            opcode = opcode.replace('+', '_')
            O.__name__ = opcode
            globals()[opcode] = O

There's also a performance hack in there: "given:" drops you down into
a function scope, so you get the benefits of function local
performance.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From p.f.moore at gmail.com  Mon Apr 11 12:12:05 2011
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 11 Apr 2011 11:12:05 +0100
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTinMC+fiyuWb8p-NEiiQVciKT=yttw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<BANLkTinMC+fiyuWb8p-NEiiQVciKT=yttw@mail.gmail.com>
Message-ID: <BANLkTi=hbqJjOgBM9f5sB5SnTVOn-hPa0w@mail.gmail.com>

On 11 April 2011 05:10, Eugene Toder <eltoder at gmail.com> wrote:
> On Sun, Apr 10, 2011 at 11:03 PM, Guido van Rossum <guido at python.org> wrote:
>> Your continued insistence on hearing an explanation why it is missing
>> is beginning to sound more and more like asking when Python stopped
>> beating his wife.
>
> Sorry, I don't get the reference.

The reference is to the "canonical" unanswerable question - "When did
you stop beating your wife?" There is no good answer because the
question is based on an incorrect assumption.

> On Sun, Apr 10, 2011 at 11:03 PM, Guido van Rossum <guido at python.org> wrote:
>> (This is not unusual. We get a lot of questions phrased as "why does
>> Python not do X?" Do you think that is a good way to get a discussion
>> started?)
>
> In my defense, I'm actually more interested in the history than in
> adding this feature, as I find the lack of it a pretty small niggle.

As far as history goes, the answer is basically "because no-one
thought of it (or if they did, they didn't care about it enough to
implement it)". You seem to be insisting that there has to be a deeper
reason, though - which is why the discussion of history/motivation is
becoming frustrating - there really isn't anything deeper.

> The discussion was started with specific examples and suggestions, but
> it was not very interesting so far, because many (not all) responses
> are a generic 'new features are bad because they are new' and it's
> very hard to argue with that :-) And I do not dispute the fact that
> new features need a high level of scrutiny.

The *only* interesting part of this thread to me is the discussion
around something like "given" (PEP 3150, which I'd forgotten about).
Oh, and some of the philosophical discussions on readability...

Paul.


From guido at python.org  Mon Apr 11 16:57:30 2011
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Apr 2011 07:57:30 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikAv8tzo19kbVHsxOw6ggk--foVTQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<BANLkTimn=BO+fr1GPDTKHDaXvtVciZrhcQ@mail.gmail.com>
	<BANLkTikAv8tzo19kbVHsxOw6ggk--foVTQ@mail.gmail.com>
Message-ID: <BANLkTi=gwVuBpRjv_i58nt6N4esZwM9KAw@mail.gmail.com>

On Sun, Apr 10, 2011 at 9:20 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Mon, Apr 11, 2011 at 3:42 AM, Guido van Rossum <guido at python.org> wrote:
>> OTOH I am personally a big fan of PEP 3150; now that the moratorium is
>> lifted I hope that someone will attempt a proper implementation of it
>> so we can find out all the odd corner cases (which I'm sure will be
>> plentiful given that this is quite a novel thing for Python).
>
> Not *quite* as novel as one might first think - the specific proposal
> I currently have in there has its roots in the way list/set/dict
> comprehensions work under the hood (in CPython, anyway). So we know
> the core principle is sound, although I'm sure you're correct that
> there are more corner cases still to be found in generalising the
> comprehension-style copy-in/copy-out semantics appropriately.
>
> I also thought of a somewhat decent use case for the feature, too:
> "builder" code in module and class namespaces, where you need some
> temporary functions, variables and loops to create a named variable,
> but then have to make sure to clean up your temporary storage
> locations to avoid polluting the public namespace. In the past, I'd
> mostly been thinking in terms of function namespaces, and keeping
> those "clean" is nowhere near as important as the situation for
> classes and modules (it's quite likely someone else has brought this
> up in the past, .
>
> To lift some examples from
> https://bitbucket.org/pypy/pypy/src/default/lib_pypy/disassembler.py
> (which is what got me thinking along these lines)
>
> from opcode import __all__ as _opcodes_all
> __all__ = ["dis","disassemble","distb","disco"] + _opcodes_all
> del _opcodes_all
>
> Could become:
>
> __all__ = ["dis","disassemble","distb","disco"] + opcode.__all__ given:
> ? ?import opcode

I like this one, but I'd like to see some more local examples too.

> And:
>
> def _setup():
> ? ?for opcode in opname:
> ? ? ? ?if not opcode.startswith('<'):
> ? ? ? ? ? ?class O(Opcode):
> ? ? ? ? ? ? ? ?pass
> ? ? ? ? ? ?opcode = opcode.replace('+', '_')
> ? ? ? ? ? ?O.__name__ = opcode
> ? ? ? ? ? ?globals()[opcode] = O
>
> _setup()
>
> Could become:
>
> pass given: # I'll add "pass" to the list of supported statements in PEP 3150
> ? ?for opcode in opname:
> ? ? ? ?if not opcode.startswith('<'):
> ? ? ? ? ? ?class O(Opcode):
> ? ? ? ? ? ? ? ?pass
> ? ? ? ? ? ?opcode = opcode.replace('+', '_')
> ? ? ? ? ? ?O.__name__ = opcode
> ? ? ? ? ? ?globals()[opcode] = O
>
> There's also a performance hack in there: "given:" drops you down into
> a function scope, so you get the benefits of function local
> performance.

Can't say I like this one. "pass given" sounds icky (and I think
earlier in this thread someone flagged it as undesirable).

I think that "given" will particularly shine in code written in a
"top-down" programming style, where one first presents the high-level
outcome and pushes details to the back. This can currently be done
quite well by putting the "main()" function definition first and then
developing it from there, but there is a certain elegance in having
the helper functions not exposed at the module level. (Although I can
also see that people who really like this style will overuse it and
put everything in a big sprawling deeply-nested structure, as opposed
to making the helpers all siblings. And it won't increase
testability.)

-- 
--Guido van Rossum (python.org/~guido)


From greg.ewing at canterbury.ac.nz  Tue Apr 12 01:07:37 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 12 Apr 2011 11:07:37 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=HAQSsZQKBm0GzT-TfR07edmtp9g@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTi=HAQSsZQKBm0GzT-TfR07edmtp9g@mail.gmail.com>
Message-ID: <4DA389B9.7080905@canterbury.ac.nz>

Bruce Leban wrote:

>     ( for {seq} + 1 )   =>  ( x + 1 for x in seq )
>     ( for f({seq}) )    =>  ( f(x) for x in seq )
>     ( if {seq} > 1 }    =>  ( x for x in seq if x > 1 )

I don't think there's anything wrong whatsoever with the
current way of doing the first two of these.

The third one could perhaps benefit from something like

     (all x in seq if ...)

but I tend to agree that the gain is too small to be worth
a change.

> C# the lambda syntax is terse 
> and I find it unreadable if written without line breaks:
> 
>     f(x, y + 1, x => x + 1, y + 1)

Seems to me the readability problem there is mainly due to it
being mixed in with other comma-separated expressions. I'm not
sure it's really all that much better with lambda:

    f(x, y + 1, lambda x: x + 1, y + 1)

I'm not even sure offhand how that parses, so I'd probably put
parens around the lambda anyway to be sure:

    f(x, y + 1, (lambda x: x + 1), y + 1)

Now do that with the terse version:

    f(x, y + 1, (x => x + 1), y + 1)

This doesn't look significantly worse to me than any other
function call with operators in the argument expressions.

If there's a problem spotting the fact that there's an =>
in there, it could be made a little more noticeable:

    f(x, y + 1, (x ==> x + 1), y + 1)

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Apr 12 01:17:28 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 12 Apr 2011 11:17:28 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
Message-ID: <4DA38C08.7060308@canterbury.ac.nz>

Nick Coghlan wrote:

> ys = [y for x in xs given f(x) as y if y]

Hmmm. Here I find myself getting tripped up by the ordering
when I try to read that smoothly.

I think it's because words like 'given' or 'where', as used
in mathematics, imply a definition of something that's been
used *before*, whereas here it's defining something to be
used *after* (i.e. in the following 'if' clause).

Mathematicians use 'let' when they're defining something to
be used subsequently. So it should really be either

   [y for x in xs letting y = f(x) if y]

or

   [y for x in xs if y given y = f(x)]

Incidentally, I think this also demonstrates that proposing
a new syntax with a <new-keyword-goes-here> placeholder doesn't
really work -- usually the choice of keyword matters, and it
has an effect on how the rest of the syntax should be arranged.
Or at least it does to someone with a strong sense of language
design aesthetics.

-- 
Greg



From greg.ewing at canterbury.ac.nz  Tue Apr 12 01:24:54 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 12 Apr 2011 11:24:54 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTin5Fdm3havXWM-XLBws+Zpscw0yxQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<1302495056.18762.4.camel@localhost.localdomain>
	<BANLkTin5Fdm3havXWM-XLBws+Zpscw0yxQ@mail.gmail.com>
Message-ID: <4DA38DC6.3080506@canterbury.ac.nz>

Eugene Toder wrote:
> The problem with 'where'
> is that it's already used as identifier by various SQL frameworks,
> which is why PEP 3150 uses 'given' instead.

That's not necessarily a deal-breaker -- it could be made
context-sensitive, like was done with 'as' for a few versions
before it became a true keyword.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Apr 12 01:32:52 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 12 Apr 2011 11:32:52 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikAv8tzo19kbVHsxOw6ggk--foVTQ@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<BANLkTimn=BO+fr1GPDTKHDaXvtVciZrhcQ@mail.gmail.com>
	<BANLkTikAv8tzo19kbVHsxOw6ggk--foVTQ@mail.gmail.com>
Message-ID: <4DA38FA4.4030605@canterbury.ac.nz>

Nick Coghlan wrote:

> I also thought of a somewhat decent use case for the feature, too:
> "builder" code in module and class namespaces,

It would also give you a really nice way to define properties:

   foo = property(get, set) given:

     def get(self):
       ...

     def set(self, x):
       ...

and to pass thunks to functions:

   do_something(blarg, body) given:
     def body(stuff):
       ...

-- 
Greg


From ncoghlan at gmail.com  Tue Apr 12 01:46:37 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Apr 2011 09:46:37 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4DA38DC6.3080506@canterbury.ac.nz>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<1302495056.18762.4.camel@localhost.localdomain>
	<BANLkTin5Fdm3havXWM-XLBws+Zpscw0yxQ@mail.gmail.com>
	<4DA38DC6.3080506@canterbury.ac.nz>
Message-ID: <BANLkTi=C0GTuVsnVc3qu38d63hb2tmEkXw@mail.gmail.com>

On Tue, Apr 12, 2011 at 9:24 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Eugene Toder wrote:
>>
>> The problem with 'where'
>> is that it's already used as identifier by various SQL frameworks,
>> which is why PEP 3150 uses 'given' instead.
>
> That's not necessarily a deal-breaker -- it could be made
> context-sensitive, like was done with 'as' for a few versions
> before it became a true keyword.

Eugene slightly mischaracterised the objection I have to the choice of
"where" as the keyword in the context of PEP 3150. It isn't that it
breaks existing libraries, it breaks database programmers' *brains*,
because "where" is the SQL keyword for filtering.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Tue Apr 12 01:49:12 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Apr 2011 09:49:12 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4DA38C08.7060308@canterbury.ac.nz>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
Message-ID: <BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>

On Tue, Apr 12, 2011 at 9:17 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>
>> ys = [y for x in xs given f(x) as y if y]
>
> Hmmm. Here I find myself getting tripped up by the ordering
> when I try to read that smoothly.
>
> I think it's because words like 'given' or 'where', as used
> in mathematics, imply a definition of something that's been
> used *before*, whereas here it's defining something to be
> used *after* (i.e. in the following 'if' clause).

This one is tricky, since the assignment is *after* the for loop, but
*before* the filter condition.

You could reorder it as below, but the translation to long-form Python
code wouldn't be quite as clean (since the order of clauses wouldn't
follow the order of nesting any more).

ys = [y for x in xs if y given f(x) as y]

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From greg.ewing at canterbury.ac.nz  Tue Apr 12 02:10:36 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 12 Apr 2011 12:10:36 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=C0GTuVsnVc3qu38d63hb2tmEkXw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<1302495056.18762.4.camel@localhost.localdomain>
	<BANLkTin5Fdm3havXWM-XLBws+Zpscw0yxQ@mail.gmail.com>
	<4DA38DC6.3080506@canterbury.ac.nz>
	<BANLkTi=C0GTuVsnVc3qu38d63hb2tmEkXw@mail.gmail.com>
Message-ID: <4DA3987C.7060905@canterbury.ac.nz>

Nick Coghlan wrote:
> It isn't that it
> breaks existing libraries, it breaks database programmers' *brains*,
> because "where" is the SQL keyword for filtering.

The SQL usage is a special case of the more general way it's
used in mathematics. I don't think database programmers can
expect to hijack the term and force everyone else to follow
their own particular interpretation of it.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Apr 12 02:14:37 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 12 Apr 2011 12:14:37 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
Message-ID: <4DA3996D.2000106@canterbury.ac.nz>

Nick Coghlan wrote:

> This one is tricky, since the assignment is *after* the for loop, but
> *before* the filter condition.

That's why I like the 'letting' form, because it both
reads correctly and matches the order of execution.

   ys = [y for x in xs letting y = f(x) if y]

translates to

   ys = []
   for x in xs:
     y = f(x)
     if y:
       ys.append(y)

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Apr 12 02:17:34 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 12 Apr 2011 12:17:34 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4DA3987C.7060905@canterbury.ac.nz>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<1302495056.18762.4.camel@localhost.localdomain>
	<BANLkTin5Fdm3havXWM-XLBws+Zpscw0yxQ@mail.gmail.com>
	<4DA38DC6.3080506@canterbury.ac.nz>
	<BANLkTi=C0GTuVsnVc3qu38d63hb2tmEkXw@mail.gmail.com>
	<4DA3987C.7060905@canterbury.ac.nz>
Message-ID: <4DA39A1E.7010608@canterbury.ac.nz>

I wrote:
> I don't think database programmers can
> expect to hijack the term and force everyone else to follow
> their own particular interpretation of it.

Having said that, I'm starting to think that 'given' works
better the way PEP 3150 uses it, when the block contains
things other than assignments.

Somehow 'where def f' doesn't sound right, but 'given def f'
does.

-- 
Greg


From cmjohnson.mailinglist at gmail.com  Tue Apr 12 05:53:24 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Mon, 11 Apr 2011 17:53:24 -1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4DA3996D.2000106@canterbury.ac.nz>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
Message-ID: <BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>

On Mon, Apr 11, 2011 at 2:14 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> ?ys = [y for x in xs letting y = f(x) if y]

Isn't that ambiguous? What if someone used the conditional expression:
ys = [y for x in xs letting y = f(x) if y else g(x)] ? It seems like
the letting/given has to go last in order to eliminate the possible
ambiguity in the subsequent optional if clause.

-- Carl Johnson


From ncoghlan at gmail.com  Tue Apr 12 06:31:17 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Apr 2011 14:31:17 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4DA3987C.7060905@canterbury.ac.nz>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<1302495056.18762.4.camel@localhost.localdomain>
	<BANLkTin5Fdm3havXWM-XLBws+Zpscw0yxQ@mail.gmail.com>
	<4DA38DC6.3080506@canterbury.ac.nz>
	<BANLkTi=C0GTuVsnVc3qu38d63hb2tmEkXw@mail.gmail.com>
	<4DA3987C.7060905@canterbury.ac.nz>
Message-ID: <BANLkTi=NymiGF1_wcFcwuxOOV42YWdM5Eg@mail.gmail.com>

On Tue, Apr 12, 2011 at 10:10 AM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> The SQL usage is a special case of the more general way it's
> used in mathematics. I don't think database programmers can
> expect to hijack the term and force everyone else to follow
> their own particular interpretation of it.

The database programmers aren't forcing anyone to do anything. While I
don't have formal survey data to back it up, I'm reasonably confident
in stating that if you grab a random Python programmer and ask them
how they understand the term "where clause", it'll be something like:
- mostly blank looks
- many responses along the lines of the way SQL uses it
- a vanishingly small minority that are consciously aware of the
formal mathematical usage (and the way it relates to English usage of
the word)

Programming may have its origins in computer science, which in turn
has its origins in mathematics, but my personal experience is that
very few day-to-day programmers have actually had sufficient exposure
to formal mathematical terminology for it to come to mind before the
SQL use case.

Now, that personal experience is biased towards an engineering
environment, but I'd consider that more representative of the wider
industry than a university environment.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Tue Apr 12 07:06:33 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Apr 2011 15:06:33 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
Message-ID: <BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>

On Tue, Apr 12, 2011 at 1:53 PM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
> On Mon, Apr 11, 2011 at 2:14 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
>> ?ys = [y for x in xs letting y = f(x) if y]
>
> Isn't that ambiguous? What if someone used the conditional expression:
> ys = [y for x in xs letting y = f(x) if y else g(x)] ? It seems like
> the letting/given has to go last in order to eliminate the possible
> ambiguity in the subsequent optional if clause.

given/as resolves the parsing ambiguity problem by putting the names second:

ys = [y for x in xs given f(x) if p(x) else g(x) as y]

However, it does make it clear how limited such a clause still is if
it exists only at the comprehension level and isn't an expression in
its own right.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Tue Apr 12 07:23:09 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Apr 2011 15:23:09 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=gwVuBpRjv_i58nt6N4esZwM9KAw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<BANLkTimn=BO+fr1GPDTKHDaXvtVciZrhcQ@mail.gmail.com>
	<BANLkTikAv8tzo19kbVHsxOw6ggk--foVTQ@mail.gmail.com>
	<BANLkTi=gwVuBpRjv_i58nt6N4esZwM9KAw@mail.gmail.com>
Message-ID: <BANLkTikymaVx9U3xSLyAPzfN4M5tdDG-kw@mail.gmail.com>

On Tue, Apr 12, 2011 at 12:57 AM, Guido van Rossum <guido at python.org> wrote:
>> pass given: # I'll add "pass" to the list of supported statements in PEP 3150
>> ? ?for opcode in opname:
>> ? ? ? ?if not opcode.startswith('<'):
>> ? ? ? ? ? ?class O(Opcode):
>> ? ? ? ? ? ? ? ?pass
>> ? ? ? ? ? ?opcode = opcode.replace('+', '_')
>> ? ? ? ? ? ?O.__name__ = opcode
>> ? ? ? ? ? ?globals()[opcode] = O
>>
>> There's also a performance hack in there: "given:" drops you down into
>> a function scope, so you get the benefits of function local
>> performance.
>
> Can't say I like this one. "pass given" sounds icky (and I think
> earlier in this thread someone flagged it as undesirable).

It isn't on the list currently in the PEP, as the only ones I included
there were the simple statements that permitted the use of
subexpressions.

However, if "pass" is left *off* the list, then the moral equivalent
of "pass given:" could still be written in a variety of other ways,
such as:

0 given: # Somewhat counterintuitive due to "if 0:"!
  pass
1 given:
  pass
"pass" given:
  pass
... given:
  pass

I figure I may as well bite the bullet and include "pass" as an
obvious way of doing inline code in a private scope.

> I think that "given" will particularly shine in code written in a
> "top-down" programming style, where one first presents the high-level
> outcome and pushes details to the back. This can currently be done
> quite well by putting the "main()" function definition first and then
> developing it from there, but there is a certain elegance in having
> the helper functions not exposed at the module level. (Although I can
> also see that people who really like this style will overuse it and
> put everything in a big sprawling deeply-nested structure, as opposed
> to making the helpers all siblings. And it won't increase
> testability.)

Yeah, being able to reach in and explicitly test "_" prefixed helpers
is very handy in writing good white box test suites. My own feelings
on PEP 3150 still sit around the level of finding it interesting and
potentially valuable as a concept (otherwise I wouldn't have written
the PEP in the first place!), but I'm not yet convinced that its net
impact on the language would be positive. My main stumbling block is
the fact I still can't characterise suitably obvious criteria as to
when it should be used over normal imperative code. I'd be a lot
happier with the PEP if I could include a specific proposal for PEP 8
additions regarding its applicability.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Tue Apr 12 07:47:22 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Apr 2011 15:47:22 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
Message-ID: <BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>

On Tue, Apr 12, 2011 at 3:06 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> ys = [y for x in xs given f(x) if p(x) else g(x) as y]

You know, I may have spoken too soon in saying PEP 3150 couldn't help
with the list comprehension use case.

def remember(f):
    """Decorator that remembers result until arguments change"""
    # Essentially an LRU cache for exactly 1 entry
    last_args = object(), object()
    last_result = None
    @functools.wraps(f)
    def wrapped(*args, **kwds):
        nonlocal last_args, last_result
        if last_args != args, kwds:
            last_args = args, kwds
            last_value = f(*args, **kwds)
        return last_value
    return wrapped

ys = [y(x) for x in xs if y(x)] given:
    y = remember(f)

Or for the more complicated case:

ys = [y(x) for x in xs if y(x)] given:
    @remember
    def y(x):
        fx  = f(x)
        return fx if fx else g(x)

Or even (using the subgenerator approach):

ys = [y for y in all_y if y] given:
    all_y = (f(x) for x in x)

There are all *sorts* of tricks that open up once you don't even need
to think about possible impacts on the local namespace.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From g.brandl at gmx.net  Tue Apr 12 08:31:52 2011
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 12 Apr 2011 08:31:52 +0200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
Message-ID: <io0rkr$2h3$1@dough.gmane.org>

On 12.04.2011 07:06, Nick Coghlan wrote:
> On Tue, Apr 12, 2011 at 1:53 PM, Carl M. Johnson
> <cmjohnson.mailinglist at gmail.com> wrote:
>> On Mon, Apr 11, 2011 at 2:14 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>>
>>>  ys = [y for x in xs letting y = f(x) if y]
>>
>> Isn't that ambiguous? What if someone used the conditional expression:
>> ys = [y for x in xs letting y = f(x) if y else g(x)] ? It seems like
>> the letting/given has to go last in order to eliminate the possible
>> ambiguity in the subsequent optional if clause.
> 
> given/as resolves the parsing ambiguity problem by putting the names second:
> 
> ys = [y for x in xs given f(x) if p(x) else g(x) as y]

Is it just me, or do others find this "string of clauses" (no matter what
the actual keywords are) not very readable?

Georg



From ericsnowcurrently at gmail.com  Tue Apr 12 08:51:47 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Tue, 12 Apr 2011 00:51:47 -0600
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
Message-ID: <BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>

On Mon, Apr 11, 2011 at 11:47 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Tue, Apr 12, 2011 at 3:06 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > ys = [y for x in xs given f(x) if p(x) else g(x) as y]
>
> You know, I may have spoken too soon in saying PEP 3150 couldn't help
> with the list comprehension use case.
>
> def remember(f):
>    """Decorator that remembers result until arguments change"""
>    # Essentially an LRU cache for exactly 1 entry
>    last_args = object(), object()
>    last_result = None
>    @functools.wraps(f)
>    def wrapped(*args, **kwds):
>        nonlocal last_args, last_result
>        if last_args != args, kwds:
>            last_args = args, kwds
>            last_value = f(*args, **kwds)
>        return last_value
>    return wrapped
>
> ys = [y(x) for x in xs if y(x)] given:
>    y = remember(f)
>
> Or for the more complicated case:
>
> ys = [y(x) for x in xs if y(x)] given:
>    @remember
>    def y(x):
>        fx  = f(x)
>        return fx if fx else g(x)
>
> Or even (using the subgenerator approach):
>
> ys = [y for y in all_y if y] given:
>    all_y = (f(x) for x in x)
>
> There are all *sorts* of tricks that open up once you don't even need
> to think about possible impacts on the local namespace.
>
>
Like Georg just said, the given clause seems hard to read.  He was referring
to the embedded clause in the list comprehension.  I kind of feel the same
way about the PEP 3150 syntax.  I remember when I was first exposed to list
comprehensions.  At first it was hard because it was organized differently
from any other Python code.  However, I have adjusted.  Maybe a given clause
would be the same.  I'm just not sure.

That "given" after the expression keeps throwing me off.  It just looks out
of place.  Would something like the following be feasible:

c = sqrt(a*a + b*b) given:
    a = retrieve_a()
    b = retrieve_b()

becomes:

given:
    a = retrieve_a()
    b = retrieve_b()
c = sqrt(a*a + b*b)

where the given statement is effectively _decorating_ the simple statement?
 Unfortunately I am not familiar enough yet with the compiler to know what
such decoration would entail, much less if it would be entirely too
complicated.  However, I wanted to throw out there an alternative that is
maybe more readable.  It's also one I would probably find more usable for
plugging in given blocks or disabling them, much like with decorators.

I know this throws off the approach the PEP takes, but I am trying to wrap
my head around how I would use statement local namespaces.  I like how it
cleans up from the namespace those bits that only matter to the target
statement.  It does make it easier to follow to whom those statements in the
given block belong.  Anyway, this has certainly been an enlightening
discussion.  I hope it bears fruit.

-eric

Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110412/987229e4/attachment.html>

From ericsnowcurrently at gmail.com  Tue Apr 12 09:11:57 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Tue, 12 Apr 2011 01:11:57 -0600
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
Message-ID: <BANLkTimjodn=Rh1Duo-QexL+Yj3CTbqwCg@mail.gmail.com>

On Tue, Apr 12, 2011 at 12:51 AM, Eric Snow <ericsnowcurrently at gmail.com>wrote:

>
> Like Georg just said, the given clause seems hard to read.  He was
> referring to the embedded clause in the list comprehension.  I kind of feel
> the same way about the PEP 3150 syntax.  I remember when I was first exposed
> to list comprehensions.  At first it was hard because it was organized
> differently from any other Python code.  However, I have adjusted.  Maybe a
> given clause would be the same.  I'm just not sure.
>
> That "given" after the expression keeps throwing me off.  It just looks out
> of place.  Would something like the following be feasible:
>
> c = sqrt(a*a + b*b) given:
>     a = retrieve_a()
>     b = retrieve_b()
>
> becomes:
>
> given:
>     a = retrieve_a()
>     b = retrieve_b()
> c = sqrt(a*a + b*b)
>
> where the given statement is effectively _decorating_ the simple statement?
>  Unfortunately I am not familiar enough yet with the compiler to know what
> such decoration would entail, much less if it would be entirely too
> complicated.  However, I wanted to throw out there an alternative that is
> maybe more readable.  It's also one I would probably find more usable for
> plugging in given blocks or disabling them, much like with decorators.
>
> I know this throws off the approach the PEP takes, but I am trying to wrap
> my head around how I would use statement local namespaces.  I like how it
> cleans up from the namespace those bits that only matter to the target
> statement.  It does make it easier to follow to whom those statements in the
> given block belong.  Anyway, this has certainly been an enlightening
> discussion.  I hope it bears fruit.
>
> -eric
>

Of course, a decorating syntax does not help with the original scenario,
with embedded  given statements in list comprehensions.  But maybe that
embedded syntax is too hard to read.

-eric

_______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110412/12bc506e/attachment.html>

From stephen at xemacs.org  Tue Apr 12 10:01:15 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 12 Apr 2011 17:01:15 +0900
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4DA3996D.2000106@canterbury.ac.nz>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
Message-ID: <87bp0bzyno.fsf@uwakimon.sk.tsukuba.ac.jp>

Greg Ewing writes:
 > Nick Coghlan wrote:
 > 
 > > This one is tricky, since the assignment is *after* the for loop, but
 > > *before* the filter condition.

I don't have a problem with *reading* that, since you can't really
win: "y" is used both before and after the "given" binding.  I would
expect you will need a restriction against using a "given"-bound
variable in another "given" clause, so syntactically I don't think it
matters where in the expression it appears.

 > That's why I like the 'letting' form, because it both
 > reads correctly and matches the order of execution.
 > 
 >    ys = [y for x in xs letting y = f(x) if y]

I'm sorry, but I read that three times and it parsed as "y gets
undefined if it is false" every time.  This is way worse than
"x if y else z", which awkward but I can't parse it any way other than
"x (if y), else z".  For some reason "given" binds the expression a
lot more tightly than "letting" does in my dialect.  I do prefer the
"given" at the end, but putting it in the middle doesn't bother me.

It's not that I couldn't get used to it, but I suspect I would never
learn to like it.  I hope others feel the same way.<wink>


From ncoghlan at gmail.com  Tue Apr 12 10:03:30 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Apr 2011 18:03:30 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <io0rkr$2h3$1@dough.gmane.org>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<io0rkr$2h3$1@dough.gmane.org>
Message-ID: <BANLkTinEWEwOCZxAoBJ8c5ZTRRY=JnYMaw@mail.gmail.com>

On Tue, Apr 12, 2011 at 4:31 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> Is it just me, or do others find this "string of clauses" (no matter what
> the actual keywords are) not very readable?

Nope, it's not just you :)

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Tue Apr 12 10:31:47 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Apr 2011 18:31:47 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
Message-ID: <BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>

On Tue, Apr 12, 2011 at 4:51 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> I know this throws off the approach the PEP takes, but I am trying to wrap
> my head around how I would use statement local namespaces. ?I like how it
> cleans up from the namespace those bits that only matter to the target
> statement.? It does make it easier to follow to whom those statements in the
> given block belong.

Yeah, in order variants have been part of the discussion since the
beginning, usually in a 2-suite form like:

let:
   # local namespace
in:
  # binding here affect surrounding scope
  # but can also see names bound in "let" clause

Dropping the second suite (and having the given clause implicit modify
the following statement) isn't really feasible (and far more brain
bending than adding a new clause to augment simple statements).

I really don't see much point to the in-order variants, hence why
discussing them is currently a TO-DO note in the PEP rather than a
fully fleshed out description of the problems I have with them.

If I had to come up with a concise rationale for when you would use
them (with the post-fix syntax and anonymous function semantics
proposed in the PEP):

1. You wish to avoid polluting the current namespace with working
variables and helper functions (most relevant for module and class
level code, but may also be relevant for functions in some closure
related contexts)

2. You wish to provide greater prominence to the final statement in a
series of operations, making it clear to the reader that the other
statements are mere setup for that final step. This is similar to the
principle of decorators, where the important information is the
function name, parameters, annotations and applied decorators, while
the precise implementation details will be uninteresting for many
readers.

3. You want early binding for closure references without resorting to
the default argument hack (including early binding of module globals
and class variables)

4. You want to enable function-level optimisations in module or class level code

1 and 2 are the core of the rationale for even proposing the idea of
statement local namespaces in the first place, but come with the
downside of making it harder to test your setup code. 3 is one of the
aspects of my most proposed implementation strategy that I most like
(since the default-argument hack is a genuinely ugly wart). 4 is just
a perk of my proposed implementation strategy rather than a compelling
use case in its own right.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From g.brandl at gmx.net  Tue Apr 12 11:57:13 2011
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 12 Apr 2011 11:57:13 +0200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
	<BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
Message-ID: <io17ls$7r7$1@dough.gmane.org>

On 12.04.2011 10:31, Nick Coghlan wrote:
> On Tue, Apr 12, 2011 at 4:51 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> I know this throws off the approach the PEP takes, but I am trying to wrap
>> my head around how I would use statement local namespaces.  I like how it
>> cleans up from the namespace those bits that only matter to the target
>> statement.  It does make it easier to follow to whom those statements in the
>> given block belong.
> 
> Yeah, in order variants have been part of the discussion since the
> beginning, usually in a 2-suite form like:
> 
> let:
>    # local namespace
> in:
>   # binding here affect surrounding scope
>   # but can also see names bound in "let" clause
> 
> Dropping the second suite (and having the given clause implicit modify
> the following statement) isn't really feasible (and far more brain
> bending than adding a new clause to augment simple statements).

However, if dropping the second suite is allowed (without any effect on
the next statement of course), you escape from the "pass given" ugliness :)

Georg




From mal at egenix.com  Tue Apr 12 12:05:49 2011
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 12 Apr 2011 12:05:49 +0200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <io0rkr$2h3$1@dough.gmane.org>
References: <4D9FDF02.6080402@gmx.net>	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>	<201104101513.p3AFDI94025509@theraft.openend.se>	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>	<eltoder@gmail.com>	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>	<201104110039.p3B0daqp011821@theraft.openend.se>	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>	<4DA38C08.7060308@canterbury.ac.nz>	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>	<4DA3996D.2000106@canterbury.ac.nz>	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<io0rkr$2h3$1@dough.gmane.org>
Message-ID: <4DA423FD.6030802@egenix.com>

Georg Brandl wrote:
> On 12.04.2011 07:06, Nick Coghlan wrote:
>> On Tue, Apr 12, 2011 at 1:53 PM, Carl M. Johnson
>> <cmjohnson.mailinglist at gmail.com> wrote:
>>> On Mon, Apr 11, 2011 at 2:14 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>>>
>>>>  ys = [y for x in xs letting y = f(x) if y]
>>>
>>> Isn't that ambiguous? What if someone used the conditional expression:
>>> ys = [y for x in xs letting y = f(x) if y else g(x)] ? It seems like
>>> the letting/given has to go last in order to eliminate the possible
>>> ambiguity in the subsequent optional if clause.
>>
>> given/as resolves the parsing ambiguity problem by putting the names second:
>>
>> ys = [y for x in xs given f(x) if p(x) else g(x) as y]
> 
> Is it just me, or do others find this "string of clauses" (no matter what
> the actual keywords are) not very readable?

Same here.

If the "given:" blocks are only meant to hide away things in
a separate namespace, I'm not sure why we need a new keyword.

This is already possible using the "class" keyword and it even
gives the namespace a name ;-)

class myData:
  x = setup_x()
  y = setup_y()

z = myFormula(myData.x, myData.y)

That's a lot more readable, pythonic and intuitive than
the proposed "given:" block syntax, which -to me at least-
looks confusing.

BTW, I don't find the comparison to the list comprehension
syntax valid:

List comprehensions only allow
a very limited set of qualifiers which form the list
definition. As such, it makes sense to have the definition
behind the item expression (and it follows the rules used
in math for similar definitions).

The "given:" block, OTOH, makes no such restrictions
and may well contain code that doesn't have anything to
do with the statement it is supposed to configure. It has
the potential of introducing top-posting to Python code:

answer_to_question given:
  some_other_blurb
  original_question
  more_other_blurb

And things get even more exciting if you start to nest
these:

answer_to_original_question given:
  comment_to_answer_of_other_question given:
    answer_to_some_other_question given:
      other_question
    new_angle_to_discussion
  other_blurb
  original_question
  more_other_blurb

Following the flow of program execution and relevance
of the different parts to the final result can be
very confusing.

Also note that it is rather unusual and unexpected for
Python to execute the innermost block before the
outer one as it would happen in the above example.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 12 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/


From eltoder at gmail.com  Tue Apr 12 14:10:55 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Tue, 12 Apr 2011 08:10:55 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
Message-ID: <BANLkTi=vtXygc0fkDcxosSJyEWU=OK5J9A@mail.gmail.com>

On Mon, Apr 11, 2011 at 11:53 PM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
> On Mon, Apr 11, 2011 at 2:14 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
>> ?ys = [y for x in xs letting y = f(x) if y]
>
> Isn't that ambiguous? What if someone used the conditional expression:
> ys = [y for x in xs letting y = f(x) if y else g(x)] ? It seems like
> the letting/given has to go last in order to eliminate the possible
> ambiguity in the subsequent optional if clause.

This ambiguity already exists -- you can write conditional expression
in iteratee of for or in condition of if, but it won't parse, because
the conflict is resolved in favour of list comprehension's 'if'. You
have to put if/else in parenthesis.

Eugene


From eltoder at gmail.com  Tue Apr 12 14:27:03 2011
From: eltoder at gmail.com (Eugene Toder)
Date: Tue, 12 Apr 2011 08:27:03 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
Message-ID: <BANLkTimpJwkyRgEDFM7CCuxXrmW=9uv3vg@mail.gmail.com>

On Tue, Apr 12, 2011 at 1:47 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> You know, I may have spoken too soon in saying PEP 3150 couldn't help
> with the list comprehension use case.

Alternatively, comprehension can be used instead of PEP 3150, e.g.

desired_property = next(
    calc_value(temp, depth, purity, salinity, size, density)
    for sea in [water()]
    for temp in [get_temperature(sea)]
    for depth in [get_depth(sea)]
    for purity in [get_purity(sea)]
    for saltiness in [get_salinity(sea)]
    for size in [get_size(sea)]
    for density in [get_density(sea)]
)
# Further operations using desired_property

> That "given" after the expression keeps throwing me off. ?It just looks out
> of place. ?Would something like the following be feasible:
> c = sqrt(a*a + b*b) given:
> ? ? a = retrieve_a()
> ? ? b = retrieve_b()

In Haskell it's usually indented differently:

c = sqrt(a*a + b*b)
      given:
? ?       a = retrieve_a()
? ?       b = retrieve_b()

Eugene


From guido at python.org  Tue Apr 12 17:24:35 2011
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Apr 2011 08:24:35 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <io0rkr$2h3$1@dough.gmane.org>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<io0rkr$2h3$1@dough.gmane.org>
Message-ID: <BANLkTikbJM0ShUF8qhTaoLSS2n0vwAtVXA@mail.gmail.com>

On Mon, Apr 11, 2011 at 11:31 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> On 12.04.2011 07:06, Nick Coghlan wrote:
[...]
>> ys = [y for x in xs given f(x) if p(x) else g(x) as y]
>
> Is it just me, or do others find this "string of clauses" (no matter what
> the actual keywords are) not very readable?

No, I can't parse it either.

-- 
--Guido van Rossum (python.org/~guido)


From tjreedy at udel.edu  Tue Apr 12 18:09:00 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 12 Apr 2011 12:09:00 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <io0rkr$2h3$1@dough.gmane.org>
References: <4D9FDF02.6080402@gmx.net>	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>	<201104101513.p3AFDI94025509@theraft.openend.se>	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>	<eltoder@gmail.com>	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>	<201104110039.p3B0daqp011821@theraft.openend.se>	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>	<4DA38C08.7060308@canterbury.ac.nz>	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>	<4DA3996D.2000106@canterbury.ac.nz>	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<io0rkr$2h3$1@dough.gmane.org>
Message-ID: <io1tes$jel$1@dough.gmane.org>

On 4/12/2011 2:31 AM, Georg Brandl wrote:
> On 12.04.2011 07:06, Nick Coghlan wrote:

>> ys = [y for x in xs given f(x) if p(x) else g(x) as y]
>
> Is it just me,

No.

> or do others find this "string of clauses" (no matter what
> the actual keywords are) not very readable?

I think people are trying to push an optional convenience feature way 
too far. I kind of wish comprehensions had been limited to one for 
clause (the source) and an optional if clause (the filter), as they 
pretty much are in math. Of course, I am one who switched to Python 
instead of lisp, perl, etc, *because* it read like pseudocode, with a 
nice mixture of expressions within statements, each in their own 
(logical) line. (I could have written this paragraph as a single 
sentence with a string of clauses, as I have seen in both older English 
and German, but I am not a fan of  that either ;-).

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Tue Apr 12 18:35:40 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 12 Apr 2011 12:35:40 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>	<201104101513.p3AFDI94025509@theraft.openend.se>	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>	<eltoder@gmail.com>	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>	<201104110039.p3B0daqp011821@theraft.openend.se>	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>	<4DA38C08.7060308@canterbury.ac.nz>	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>	<4DA3996D.2000106@canterbury.ac.nz>	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
	<BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
Message-ID: <io1v0s$to3$1@dough.gmane.org>

On 4/12/2011 4:31 AM, Nick Coghlan wrote:

> 1. You wish to avoid polluting the current namespace with working
> variables and helper functions (most relevant for module and class
> level code, but may also be relevant for functions in some closure
> related contexts)

I am puzzled why namespace sanitation fans are so allergic to the 
namespace cleanup statement == 'del'.

> 2. You wish to provide greater prominence to the final statement in a
> series of operations, making it clear to the reader that the other
> statements are mere setup for that final step.

Final position *is* a place of prominence.

> This is similar to the
> principle of decorators, where the important information is the
> function name, parameters, annotations and applied decorators, while
> the precise implementation details will be uninteresting for many
> readers.

And decorators put the def line *last* instead of first, whereas the 
proposed 'given' moves the punchline statement from last to first.

A module or class doc string can list the important names, which should 
be readable and multiple chars. A module can reiterate with __all__. 
Private names get a leading _. Temporary names can be short and deleted 
when not needed.

-- 
Terry Jan Reedy



From alexander.belopolsky at gmail.com  Tue Apr 12 19:03:30 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Tue, 12 Apr 2011 13:03:30 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <io1v0s$to3$1@dough.gmane.org>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
	<BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
	<io1v0s$to3$1@dough.gmane.org>
Message-ID: <BANLkTinbM5x0r3rnV11=9B1mLONejw3jcw@mail.gmail.com>

On Tue, Apr 12, 2011 at 12:35 PM, Terry Reedy <tjreedy at udel.edu> wrote:
..
> I am puzzled why namespace sanitation fans are so allergic to the namespace
> cleanup statement == 'del'.

Because using cleanup may unexpectedly clobber the variables you want
to keep.  When using

for var in [..]:
  ..
del var

idiom, I tend to maintain some naming convention for what var can be
so that it does not conflict with other global variables.  Naming
conventions are usually considered suboptimal work-arounds for
languages with poor namespace support.


From jimjjewett at gmail.com  Tue Apr 12 19:16:59 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Tue, 12 Apr 2011 13:16:59 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <io1v0s$to3$1@dough.gmane.org>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
	<BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
	<io1v0s$to3$1@dough.gmane.org>
Message-ID: <BANLkTin-coFSSSxJ5d12y6F4-6RU1MdihA@mail.gmail.com>

On 4/12/11, Terry Reedy <tjreedy at udel.edu> wrote:
> On 4/12/2011 4:31 AM, Nick Coghlan wrote:

>> 1. You wish to avoid polluting the current namespace with working
>> variables and helper functions (most relevant for module and class
>> level code, but may also be relevant for functions in some closure
>> related contexts)

> I am puzzled why namespace sanitation fans are so allergic to the
> namespace cleanup statement == 'del'.

You have to do it explicitly, which calls attention to it precisely
when you were ready to forget it.

In other words, it has all the disadvantages that pre-decorator
function wrapping did, and (if it were really needed) all the
disadvantages of manual memory management, and the additional concern
that -- because it is not idiomatic -- readers will expect that
variable name (or at least that namespace) to be particularly
important.

-jJ


From guido at python.org  Tue Apr 12 19:19:03 2011
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Apr 2011 10:19:03 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTinbM5x0r3rnV11=9B1mLONejw3jcw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
	<BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
	<io1v0s$to3$1@dough.gmane.org>
	<BANLkTinbM5x0r3rnV11=9B1mLONejw3jcw@mail.gmail.com>
Message-ID: <BANLkTinZuM8uxOb6zs1STE+86nuXWo_5OQ@mail.gmail.com>

On Tue, Apr 12, 2011 at 10:03 AM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Tue, Apr 12, 2011 at 12:35 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> ..
>> I am puzzled why namespace sanitation fans are so allergic to the namespace
>> cleanup statement == 'del'.
>
> Because using cleanup may unexpectedly clobber the variables you want
> to keep. ?When using
>
> for var in [..]:
> ?..
> del var
>
> idiom, I tend to maintain some naming convention for what var can be
> so that it does not conflict with other global variables. ?Naming
> conventions are usually considered suboptimal work-arounds for
> languages with poor namespace support.

Also note that if that loop happened to be an empty loop, var would
not be set, and del var would fail.

Another reason why del is suboptimal is that if you have multiple
exits from your block you need a try/finally just to delete the
variables, but then you definitely need to deal with the possibility
of the variable not yet existing. Either way it's a mess.

-- 
--Guido van Rossum (python.org/~guido)


From jimjjewett at gmail.com  Tue Apr 12 19:25:23 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Tue, 12 Apr 2011 13:25:23 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
	<BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
Message-ID: <BANLkTin8W3gEXDGZ84ARGzotk=QU7wA2Yw@mail.gmail.com>

On 4/12/11, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Yeah, in order variants have been part of the discussion since the
> beginning, usually in a 2-suite form like:

> let:
>    # local namespace
> in:
>   # binding here affect surrounding scope
>   # but can also see names bound in "let" clause

...
> I really don't see much point to the in-order variants, hence why
> discussing them is currently a TO-DO note in the PEP rather than a
> fully fleshed out description of the problems I have with them.

If you're using two blocks (like the if statement), then why do they
have to be in order?

    exec:  # or "do" or "run" or ...
        ... # ordinary suite
    given:
        ... # suite limited to name bindings statements?  normal suite?

> 1. You wish to avoid polluting the current namespace with working
> variables and helper functions (most relevant for module and class
> level code, but may also be relevant for functions in some closure
> related contexts)

The idea of a throw-away internal class does solve this, but I admit
it looks very odd -- to the point that I would be looking for a
keyword other than class.

> 2. You wish to provide greater prominence to the final statement in a
> series of operations, making it clear to the reader that the other
> statements are mere setup for that final step. This is similar to the
> principle of decorators, where the important information is the
> function name, parameters, annotations and applied decorators, while
> the precise implementation details will be uninteresting for many
> readers.

The reverse-order double suite solves this, so long as you keep the
suites reasonably sized.

> 3. You want early binding for closure references without resorting to
> the default argument hack (including early binding of module globals
> and class variables)

Much as I like this, it sort of requires that the whole function be
wrapped in a
do ... given, which might start to look like pointless boilerplate if
people start to do it too often.

-jJ


From john.theman.connor at gmail.com  Tue Apr 12 23:42:43 2011
From: john.theman.connor at gmail.com (jac)
Date: Tue, 12 Apr 2011 14:42:43 -0700 (PDT)
Subject: [Python-ideas] Copy-on-write when forking a python process
Message-ID: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>

Hi all,
Sorry for cross posting, but I think that this group may actually be
more appropriate for this discussion.  Previous thread is at:
http://groups.google.com/group/comp.lang.python/browse_thread/thread/1df510595483b12f

I am wondering if anything can be done about the COW (copy-on-write)
problem when forking a python process.  I have found several
discussions of this problem, but I have seen no proposed solutions or
workarounds.  My understanding of the problem is that an object's
reference count is stored in the "ob_refcnt" field of the PyObject
structure itself.  When a process forks, its memory is initially not
copied. However, if any references to an object are made or destroyed
in the child process, the page in which the objects "ob_refcnt" field
is located in will be copied.
My first thought was the obvious one: make the ob_refcnt field a
pointer into an array of all object refcounts stored elsewhere.
However, I do not think that there would be a way of doing this
without adding a lot of complexity.  So my current thinking is that it
should be possible to disable refcounting for an object.  This could
be done by adding a field to PyObject named "ob_optout".  If ob_optout
is true then py_INCREF and py_DECREF will have no effect on the
object:

from refcount import optin, optout
class Foo: pass
mylist = [Foo() for _ in range(10)]
optout(mylist)  # Sets ob_optout to true
for element in mylist:
      optout(element) # Sets ob_optout to true
Fork_and_block_while_doing_stuff(mylist)
optin(mylist) # Sets ob_optout to false
for element in mylist:
      optin(element) # Sets ob_optout to false

I realize that using shared memory is a possible solution for many of
the situations one would wish to use the above solution, but I think
that there are enough situations where one wishes to use the os's cow
mechanism and is prohibited from doing so to warrant a fix.

Has anyone else looked into the COW problem?  Are there workarounds
and/or other plans to fix it?  Does the solution I am proposing sound
reasonable, or does it seem like overkill?  Does anyone see any
(technical) problems with it?

Thanks,
--jac


From greg.ewing at canterbury.ac.nz  Wed Apr 13 00:22:23 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 13 Apr 2011 10:22:23 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
Message-ID: <4DA4D09F.4050108@canterbury.ac.nz>

Nick Coghlan wrote:

> given/as resolves the parsing ambiguity problem by putting the names second:

And I don't like that. I would rather see an = sign.
We're doing an assignment, after all, so we should
make it look like one if we can.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Wed Apr 13 00:29:54 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 13 Apr 2011 10:29:54 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <87bp0bzyno.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<87bp0bzyno.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <4DA4D262.3040805@canterbury.ac.nz>

Stephen J. Turnbull wrote:

> I don't have a problem with *reading* that, since you can't really
> win: "y" is used both before and after the "given" binding.

But the "before" usage is just the usual comprehension
quirk of putting the result expression before everything
else, even though it's actually evaluated last. Once you
allow for that, it's effectively after the given/letting/
whatever.

>  >    ys = [y for x in xs letting y = f(x) if y]
> 
> I'm sorry, but I read that three times and it parsed as "y gets
> undefined if it is false" every time.

In that case, would you prefer this?

   ys = [y for x in xs if y given y = f(x)]

-- 
Greg


From greg.ewing at canterbury.ac.nz  Wed Apr 13 00:30:01 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 13 Apr 2011 10:30:01 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
Message-ID: <4DA4D269.7030306@canterbury.ac.nz>

Carl M. Johnson wrote:

> Isn't that ambiguous? What if someone used the conditional expression:
> ys = [y for x in xs letting y = f(x) if y else g(x)] ?

No more ambiguous than 'if' already is in a comprehension,
and that's resolved by requiring parens if one of the
clauses in the comprehension contains an 'if' expression.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Wed Apr 13 00:34:29 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 13 Apr 2011 10:34:29 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4DA423FD.6030802@egenix.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<io0rkr$2h3$1@dough.gmane.org> <4DA423FD.6030802@egenix.com>
Message-ID: <4DA4D375.1040509@canterbury.ac.nz>

M.-A. Lemburg wrote:

> This is already possible using the "class" keyword
> 
> class myData:
>   x = setup_x()
>   y = setup_y()

Only for certain values of "possible". It potentially
confuses the reader by using a class in an unusual way.
When one sees a class definition, one's first thought
is that instances of it are going to be created at some
point, but that's not the case here.

-- 
Greg


From greg at krypto.org  Wed Apr 13 03:12:52 2011
From: greg at krypto.org (Gregory P. Smith)
Date: Tue, 12 Apr 2011 18:12:52 -0700
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
Message-ID: <BANLkTi=9iExR5aV6N7NHWzOj74qBB__8xw@mail.gmail.com>

On Tue, Apr 12, 2011 at 2:42 PM, jac <john.theman.connor at gmail.com> wrote:
> Hi all,
> Sorry for cross posting, but I think that this group may actually be
> more appropriate for this discussion. ?Previous thread is at:
> http://groups.google.com/group/comp.lang.python/browse_thread/thread/1df510595483b12f
>
> I am wondering if anything can be done about the COW (copy-on-write)
> problem when forking a python process. ?I have found several
> discussions of this problem, but I have seen no proposed solutions or
> workarounds. ?My understanding of the problem is that an object's
> reference count is stored in the "ob_refcnt" field of the PyObject
> structure itself. ?When a process forks, its memory is initially not
> copied. However, if any references to an object are made or destroyed
> in the child process, the page in which the objects "ob_refcnt" field
> is located in will be copied.
> My first thought was the obvious one: make the ob_refcnt field a
> pointer into an array of all object refcounts stored elsewhere.
> However, I do not think that there would be a way of doing this
> without adding a lot of complexity. ?So my current thinking is that it
> should be possible to disable refcounting for an object. ?This could
> be done by adding a field to PyObject named "ob_optout". ?If ob_optout
> is true then py_INCREF and py_DECREF will have no effect on the
> object:
>
> from refcount import optin, optout
> class Foo: pass
> mylist = [Foo() for _ in range(10)]
> optout(mylist) ?# Sets ob_optout to true
> for element in mylist:
> ? ? ?optout(element) # Sets ob_optout to true
> Fork_and_block_while_doing_stuff(mylist)
> optin(mylist) # Sets ob_optout to false
> for element in mylist:
> ? ? ?optin(element) # Sets ob_optout to false
>
> I realize that using shared memory is a possible solution for many of
> the situations one would wish to use the above solution, but I think
> that there are enough situations where one wishes to use the os's cow
> mechanism and is prohibited from doing so to warrant a fix.
>
> Has anyone else looked into the COW problem? ?Are there workarounds
> and/or other plans to fix it? ?Does the solution I am proposing sound
> reasonable, or does it seem like overkill? ?Does anyone see any
> (technical) problems with it?

I do not think most people consider this a problem.  For

Reference counting in the first place... now that is a problem.  We
shouldn't be doing it and instead should use a more modern scalable
form of garbage collection...  Immutable hashable objects in Python
(or is it just strings?) can be interned using the intern() call.
This means they will never be freed.  But I do not believe the current
implementation of interning prevents reference counting, it just adds
them to an internal map of things (ie: one final reference) so they'll
never be freed.

The biggest drawback is one you can experiment with yourself.

Py_INCREF and Py_DECREF are currently very simple.  Adding a special
case means you'd be adding an additional conditional check every time
they are called (regardless of if it is a special magic high reference
count or a new field with a bit set indicating that reference counting
is disabled for a given object).

To find out if it is worth it, try adding code that does that and
running the python benchmarks and see what happens.

I like your idea of the refcount table being stored elsewhere to
improve this particular copy on write issue but I don't really see it
as a problem a lot of people are encountering.  Got data otherwise
(obviously you are running into it... who else?)?  I do not expect
most people to fork() other than using the subprocess module where its
followed by an exec().

-gps


From mikegraham at gmail.com  Wed Apr 13 03:32:01 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Tue, 12 Apr 2011 21:32:01 -0400
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <BANLkTi=9iExR5aV6N7NHWzOj74qBB__8xw@mail.gmail.com>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<BANLkTi=9iExR5aV6N7NHWzOj74qBB__8xw@mail.gmail.com>
Message-ID: <BANLkTimiBbJ_1BgePJNYKxagtyROaUtAAA@mail.gmail.com>

On Tue, Apr 12, 2011 at 9:12 PM, Gregory P. Smith <greg at krypto.org> wrote:
> I do not think most people consider this a problem. ?For
>
> Reference counting in the first place... now that is a problem. ?We
> shouldn't be doing it and instead should use a more modern scalable
> form of garbage collection... ?Immutable hashable objects in Python
> (or is it just strings?) can be interned using the intern() call.
> This means they will never be freed. ?But I do not believe the current
> implementation of interning prevents reference counting, it just adds
> them to an internal map of things (ie: one final reference) so they'll
> never be freed.
>
> ...
>
> -gps

Python interns some strings and small ints. The intern builtin ensures
a string is in the former cache and isn't applicable for other
objects; Python automatically interns strings that look like
identifiers and you should never use the intern function yourself.

These optimizations have nothing to do with reference counting and
could be applicable under other garbage collection schemes. Reference
counting doesn't mean that interned objects can never be freed; are
you familiar with the idea of weak references?

Reference counting is a pleasantly simple though somewhat outdated
scheme. It is not nearly as limiting as I think you imagine it to be.

Mike


From stephen at xemacs.org  Wed Apr 13 04:27:08 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 13 Apr 2011 11:27:08 +0900
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4DA4D262.3040805@canterbury.ac.nz>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<87bp0bzyno.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4DA4D262.3040805@canterbury.ac.nz>
Message-ID: <878vvezy0z.fsf@uwakimon.sk.tsukuba.ac.jp>

Greg Ewing writes:
 > Stephen J. Turnbull wrote:
 > 
 > > I don't have a problem with *reading* that, since you can't really
 > > win: "y" is used both before and after the "given" binding.
 > 
 > But

Turns out to be precisely why it's readable, I think.  But the "but"
is not the point.  The point is that (in my dialect) it *is* readable.

 > >  >    ys = [y for x in xs letting y = f(x) if y]
 > > 
 > > I'm sorry, but I read that three times and it parsed as "y gets
 > > undefined if it is false" every time.
 > 
 > In that case, would you prefer this?
 > 
 >    ys = [y for x in xs if y given y = f(x)]

Yes, very much.  I would also prefer

    ys = [y for x in xs given y = f(x) if y]

to the "letting" version, though that is harder to read (for me) than
the version with the assignment at the end of the expression.

I think the difference is that "let" is *local* to the comprehension
and local context will bind to it.  "given" is a meta-word, it refers
to something *outside* the comprehension, in my dialect.  So it is
undisturbed by the local context.

YMMV, that may be very specific to me.  As I say, I could probably get
used to "letting" if most people prefer it.  But in my personal usage,
"given" is clearly better.





From cmjohnson.mailinglist at gmail.com  Wed Apr 13 05:17:06 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Tue, 12 Apr 2011 17:17:06 -1000
Subject: [Python-ideas] PEP-3150
Message-ID: <BANLkTi=4V7r=vhk9cb8ckmfQbHaFSdfw1A@mail.gmail.com>

Can we start a new thread just for discussion of PEP-3150? I
personally don't like any of the proposals to add assignment to list
comprehensions/generator expressions and I think that no matter how
it's spelt the idea is a misfeature since there's no need to cram more
info into a one-liner list comp/gen ex, but I do find the given-clause
very interesting and potentially useful. In terms of positives for the
feature, assignment in a list comp/gen ex has the potential to save a
few characters and maybe be more readable (although I haven't seen any
proposal that strikes me as more readable yet), but PEP-3150 has the
four different benefits listed previously by Nick, most notably that
it creates a temporary namespace (a honking great idea!). So, it
strikes me that the two issues are different enough that it might be
useful to separate the discussions out.


From tjreedy at udel.edu  Wed Apr 13 05:40:02 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 12 Apr 2011 23:40:02 -0400
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <BANLkTimiBbJ_1BgePJNYKxagtyROaUtAAA@mail.gmail.com>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>	<BANLkTi=9iExR5aV6N7NHWzOj74qBB__8xw@mail.gmail.com>
	<BANLkTimiBbJ_1BgePJNYKxagtyROaUtAAA@mail.gmail.com>
Message-ID: <io35uc$dk4$1@dough.gmane.org>

On 4/12/2011 9:32 PM, Mike Graham wrote:

> Python interns some strings and small ints. The intern builtin ensures

intern is deprecated in 2.7 and gone in 3.x.

> a string is in the former cache and isn't applicable for other
> objects; Python automatically interns strings that look like
> identifiers and you should never use the intern function yourself.
>
> These optimizations have nothing to do with reference counting and
> could be applicable under other garbage collection schemes. Reference
> counting doesn't mean that interned objects can never be freed; are
> you familiar with the idea of weak references?

"Changed in version 2.3: Interned strings are not immortal (like they 
used to be in Python 2.2 and before); you must keep a reference to the 
return value of intern() around to benefit from it."

-- 
Terry Jan Reedy



From ncoghlan at gmail.com  Wed Apr 13 07:53:54 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 13 Apr 2011 15:53:54 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <io1v0s$to3$1@dough.gmane.org>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
	<BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
	<io1v0s$to3$1@dough.gmane.org>
Message-ID: <BANLkTin5+T6qNNeZii8w0T93gH2=OsjyPg@mail.gmail.com>

On Wed, Apr 13, 2011 at 2:35 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 4/12/2011 4:31 AM, Nick Coghlan wrote:
>
>> 1. You wish to avoid polluting the current namespace with working
>> variables and helper functions (most relevant for module and class
>> level code, but may also be relevant for functions in some closure
>> related contexts)
>
> I am puzzled why namespace sanitation fans are so allergic to the namespace
> cleanup statement == 'del'.

For one of the key reasons decorators and context managers exist -
it's hard to audit "at the end" stuff for correctness.

>> This is similar to the
>> principle of decorators, where the important information is the
>> function name, parameters, annotations and applied decorators, while
>> the precise implementation details will be uninteresting for many
>> readers.
>
> And decorators put the def line *last* instead of first, whereas the
> proposed 'given' moves the punchline statement from last to first.

This whole discussion has been really useful to me in crystallising
*why* I see value in PEP 3150, and it is directly related to the way
function and class definitions work.

I have the glimmerings of a rewrite of PEP 3150 kicking around in my
skull, that may include restricting it to assignment statements (I'm
not 100% decided on that point as yet - I'll make up my mind as the
rest of the rewrite takes shape). The reason I am considering such a
restriction is that the new Rationale section will likely be along the
following lines:

=========================

Function and class statements in Python have a unique property
relative to ordinary assignment statements: to some degree, they are
*declarative*. They present the reader of the code with some critical
information about a name that is about to be defined, before
proceeding on with the details of the actual definition in the
function or class body.

The *name* of the object being declared is the first thing stated
after the keyword. Other important information is also given the
honour of preceding the implementation details:

- decorators (which can greatly affect the behaviour of the created
object, and were placed ahead of even the keyword and name as a matter
of practicality moreso than aesthetics)
- the docstring (on the first line immediately following the header line)
- parameters, default values and annotations for function definitions
- parent classes, metaclass and optionally other details (depending on
the metaclass) for class definitions

This PEP proposes to make a similar declarative style available for
arbitrary assignment operations, by permitting the inclusion of a
"given" suite following any simple (non-augmented) assignment
statement::

    TARGET = [TARGET2 = ... TARGETN =] EXPR given:
        SUITE

By convention, code in the body of the suite should be oriented solely
towards correctly defining the assignment operation carried out in the
header line. The header line operation should also be adequately
descriptive (e.g. through appropriate choices of variable names) to
give a reader a reasonable idea of the purpose of the operation
without reading the body of the suite.

=========================

Another addition I am considering is the idea of allowing the "given"
suite to contain a docstring, thus providing a way to make it easy to
attach a __doc__ attribute to arbitrary targets. This may require
disallowing tuple unpacking and multiple assignment targets when using
the given clause, or else simply raising a syntax error if a docstring
is present for an assignment using either of those forms.

Other use cases will of course still be possible, but that will be the
driving force behind the revised PEP.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Wed Apr 13 08:17:03 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 13 Apr 2011 16:17:03 +1000
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
Message-ID: <BANLkTi=NDbSW96vVs7rwYrMFS9KfJu3_mA@mail.gmail.com>

On Wed, Apr 13, 2011 at 7:42 AM, jac <john.theman.connor at gmail.com> wrote:
> Has anyone else looked into the COW problem? ?Are there workarounds
> and/or other plans to fix it? ?Does the solution I am proposing sound
> reasonable, or does it seem like overkill? ?Does anyone see any
> (technical) problems with it?

There's a clear workaround for the COW problem these days: use PyPy
instead of CPython :)

Currently that workaround comes at a potentially high cost in
compatibility with 3rd party C extensions, but that situation will
naturally improve over time. Given that a lot of those compatibility
problems arise *because* PyPy doesn't use refcounting natively, it's
highly unlikely that there will be any significant tinkering with
CPython's own approach.

As far as technical problems go, opting out of memory management is a
beautiful way to shoot yourself in the foot with memory leaks. All it
takes is one optout() without a corresponding optin() and an arbitrary
amount of memory may fail to be released. For example, in your own
post, any exception in Fork_and_block_while_doing_stuff() means
anything referenced directly or indirectly from mylist will be left
hanging around in memory until the process terminates. That's a *far*
worse problem than being unable to readily share memory between
processes.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From mwm at mired.org  Wed Apr 13 10:34:11 2011
From: mwm at mired.org (Mike Meyer)
Date: Wed, 13 Apr 2011 04:34:11 -0400
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
Message-ID: <20110413043411.028147f7@bhuda.mired.org>

On Tue, 12 Apr 2011 14:42:43 -0700 (PDT)
jac <john.theman.connor at gmail.com> wrote:

> Hi all,
> Sorry for cross posting, but I think that this group may actually be
> more appropriate for this discussion.  Previous thread is at:
> http://groups.google.com/group/comp.lang.python/browse_thread/thread/1df510595483b12f
> 
> I am wondering if anything can be done about the COW (copy-on-write)
> problem when forking a python process.  I have found several
> discussions of this problem, but I have seen no proposed solutions or
> workarounds.  My understanding of the problem is that an object's
> reference count is stored in the "ob_refcnt" field of the PyObject
> structure itself.  When a process forks, its memory is initially not
> copied. However, if any references to an object are made or destroyed
> in the child process, the page in which the objects "ob_refcnt" field
> is located in will be copied.

This smells like premature optimization to me. You're worried about
the kernel copying a few extra pages of user data when you're dealing
with a dictionary that's gigabytes in size. Sounds like any possibly
memory savings here would be much smaller than those that could come
from improving the data encoding.

But maybe it's not premature. Do you have measurements that show how
much extra swap space is taken up by COW copies caused by changing
reference counts in your application?

      <mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/consulting.html
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org


From solipsis at pitrou.net  Wed Apr 13 13:58:15 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Apr 2011 13:58:15 +0200
Subject: [Python-ideas] Copy-on-write when forking a python process
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<BANLkTi=9iExR5aV6N7NHWzOj74qBB__8xw@mail.gmail.com>
	<BANLkTimiBbJ_1BgePJNYKxagtyROaUtAAA@mail.gmail.com>
	<io35uc$dk4$1@dough.gmane.org>
Message-ID: <20110413135815.1fa4bbcc@pitrou.net>

On Tue, 12 Apr 2011 23:40:02 -0400
Terry Reedy <tjreedy at udel.edu> wrote:
> On 4/12/2011 9:32 PM, Mike Graham wrote:
> 
> > Python interns some strings and small ints. The intern builtin ensures
> 
> intern is deprecated in 2.7 and gone in 3.x.

It's now called sys.intern().

> > a string is in the former cache and isn't applicable for other
> > objects; Python automatically interns strings that look like
> > identifiers and you should never use the intern function yourself.
> >
> > These optimizations have nothing to do with reference counting and
> > could be applicable under other garbage collection schemes. Reference
> > counting doesn't mean that interned objects can never be freed; are
> > you familiar with the idea of weak references?
> 
> "Changed in version 2.3: Interned strings are not immortal (like they 
> used to be in Python 2.2 and before); you must keep a reference to the 
> return value of intern() around to benefit from it."

That's a rather strange sentence, because interned strings *are*
immortal (until the interpreter is shutdown).

Regards

Antoine.




From jimjjewett at gmail.com  Wed Apr 13 15:25:03 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 13 Apr 2011 09:25:03 -0400
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <878vvezy0z.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<87bp0bzyno.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4DA4D262.3040805@canterbury.ac.nz>
	<878vvezy0z.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <BANLkTimW5=4bMT4R19cc3GNiXArm3hz52A@mail.gmail.com>

On 4/12/11, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Greg Ewing writes:
>  > Stephen J. Turnbull wrote:

>  > >  >    ys = [y for x in xs letting y = f(x) if y]

>  > > I'm sorry, but I read that three times and it parsed as "y gets
>  > > undefined if it is false" every time.

Me too ... but then I try to figure out what to do with an undefined,
so the eventual answer is either an error, or what you really wanted
in the first place.

>  > In that case, would you prefer this?

>  >    ys = [y for x in xs if y given y = f(x)]

I would ... but I don't actually *like* it, I just find it less problematic.

> Yes, very much.  I would also prefer
>
>     ys = [y for x in xs given y = f(x) if y]

> to the "letting" version, though that is harder to read (for me) than
> the version with the assignment at the end of the expression.

I find this less objectionable still.  It improves more if the "given"
clause is on a separate line.

I'm not sure there *is* a likable way to put the comprehension, the
temporary assignment, and the filter into one unbroken thought.

-jJ


From jimjjewett at gmail.com  Wed Apr 13 15:43:16 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 13 Apr 2011 09:43:16 -0400
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
Message-ID: <BANLkTimdyeJPiDnB_gK+Cv+ZXa66gCfMpQ@mail.gmail.com>

On 4/12/11, jac <john.theman.connor at gmail.com> wrote:
> ... an object's
> reference count is stored in the "ob_refcnt" field of the PyObject
> structure itself.  When a process forks, its memory is initially not
> copied. However, if any references to an object are made or destroyed
> in the child process, the page in which the objects "ob_refcnt" field
> is located in will be copied.

This also causes some problems in a single process attempting to run
on multiple cores, because that change invalidates the cache.

> My first thought was the obvious one: make the ob_refcnt field a
> pointer into an array of all object refcounts stored elsewhere.

Good thought, and probably needed for some types of parallelism.

The problem is that it also means that actually using the object will
require loading from at least two memory areas -- one to update the
reference count, the other for the object itself, which may or may not
be changed.  For relatively small objects, you would effectively be
cutting your cache size in half, in addition to the new calculations.
It takes a lot of benefit for that to pay back, and it may be simpler
to just go with PyPy and an alternate memory management scheme.

-jJ


From jimjjewett at gmail.com  Wed Apr 13 15:47:24 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 13 Apr 2011 09:47:24 -0400
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <20110413135815.1fa4bbcc@pitrou.net>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<BANLkTi=9iExR5aV6N7NHWzOj74qBB__8xw@mail.gmail.com>
	<BANLkTimiBbJ_1BgePJNYKxagtyROaUtAAA@mail.gmail.com>
	<io35uc$dk4$1@dough.gmane.org> <20110413135815.1fa4bbcc@pitrou.net>
Message-ID: <BANLkTikQQ2FAFTasZ9Wt_dH9w+gwHJHMyA@mail.gmail.com>

On 4/13/11, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 12 Apr 2011 23:40:02 -0400
> Terry Reedy <tjreedy at udel.edu> wrote:

>> "Changed in version 2.3: Interned strings are not immortal (like they
>> used to be in Python 2.2 and before); you must keep a reference to the
>> return value of intern() around to benefit from it."

> That's a rather strange sentence, because interned strings *are*
> immortal (until the interpreter is shutdown).

The purpose of that change (which may no longer be effective; I
haven't checked recently) was that they were no longer immortal.  If
the last reference outside the intern dictionary was removed, then the
string was removed from the intern dictionary as well.  Intern was a
way to de-duplicate, but it didn't (by itself) make anything immortal.

-jJ


From mikegraham at gmail.com  Wed Apr 13 16:06:27 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Wed, 13 Apr 2011 10:06:27 -0400
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <20110413135815.1fa4bbcc@pitrou.net>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<BANLkTi=9iExR5aV6N7NHWzOj74qBB__8xw@mail.gmail.com>
	<BANLkTimiBbJ_1BgePJNYKxagtyROaUtAAA@mail.gmail.com>
	<io35uc$dk4$1@dough.gmane.org> <20110413135815.1fa4bbcc@pitrou.net>
Message-ID: <BANLkTikQMZTNDSZdUn8ippm=PQxMdAbawQ@mail.gmail.com>

On Wed, Apr 13, 2011 at 7:58 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 12 Apr 2011 23:40:02 -0400
> Terry Reedy <tjreedy at udel.edu> wrote:
>> "Changed in version 2.3: Interned strings are not immortal (like they
>> used to be in Python 2.2 and before); you must keep a reference to the
>> return value of intern() around to benefit from it."
>
> That's a rather strange sentence, because interned strings *are*
> immortal (until the interpreter is shutdown).
>
> Regards
>
> Antoine.

Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> x = intern("asdfqqqq")
>>> id(x)
3078566112L
>>> id("asdfqqqq")
3078566112L
>>> x = []
>>> y = []
>>> id("asdfqqqq")
3078566208L


Does not this suggest otherwise?

Mike


From solipsis at pitrou.net  Wed Apr 13 16:14:57 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Apr 2011 16:14:57 +0200
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <BANLkTikQQ2FAFTasZ9Wt_dH9w+gwHJHMyA@mail.gmail.com>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<BANLkTi=9iExR5aV6N7NHWzOj74qBB__8xw@mail.gmail.com>
	<BANLkTimiBbJ_1BgePJNYKxagtyROaUtAAA@mail.gmail.com>
	<io35uc$dk4$1@dough.gmane.org> <20110413135815.1fa4bbcc@pitrou.net>
	<BANLkTikQQ2FAFTasZ9Wt_dH9w+gwHJHMyA@mail.gmail.com>
Message-ID: <1302704097.3565.1.camel@localhost.localdomain>

Le mercredi 13 avril 2011 ? 09:47 -0400, Jim Jewett a ?crit :
> On 4/13/11, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Tue, 12 Apr 2011 23:40:02 -0400
> > Terry Reedy <tjreedy at udel.edu> wrote:
> 
> >> "Changed in version 2.3: Interned strings are not immortal (like they
> >> used to be in Python 2.2 and before); you must keep a reference to the
> >> return value of intern() around to benefit from it."
> 
> > That's a rather strange sentence, because interned strings *are*
> > immortal (until the interpreter is shutdown).
> 
> The purpose of that change (which may no longer be effective; I
> haven't checked recently) was that they were no longer immortal.  If
> the last reference outside the intern dictionary was removed, then the
> string was removed from the intern dictionary as well.  Intern was a
> way to de-duplicate, but it didn't (by itself) make anything immortal.

They're de-facto immortal, since the user can't access the intern
dictionary to remove these strings. That sentence looks like a very
misleading way of explaining an implementation detail and making it look
like a user-visible semantic change.

Regards

Antoine.




From solipsis at pitrou.net  Wed Apr 13 16:19:02 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Apr 2011 16:19:02 +0200
Subject: [Python-ideas] Copy-on-write when forking a python process
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<BANLkTi=9iExR5aV6N7NHWzOj74qBB__8xw@mail.gmail.com>
	<BANLkTimiBbJ_1BgePJNYKxagtyROaUtAAA@mail.gmail.com>
	<io35uc$dk4$1@dough.gmane.org> <20110413135815.1fa4bbcc@pitrou.net>
	<BANLkTikQMZTNDSZdUn8ippm=PQxMdAbawQ@mail.gmail.com>
Message-ID: <20110413161902.6f567070@pitrou.net>

On Wed, 13 Apr 2011 10:06:27 -0400
Mike Graham <mikegraham at gmail.com> wrote:

> On Wed, Apr 13, 2011 at 7:58 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Tue, 12 Apr 2011 23:40:02 -0400
> > Terry Reedy <tjreedy at udel.edu> wrote:
> >> "Changed in version 2.3: Interned strings are not immortal (like they
> >> used to be in Python 2.2 and before); you must keep a reference to the
> >> return value of intern() around to benefit from it."
> >
> > That's a rather strange sentence, because interned strings *are*
> > immortal (until the interpreter is shutdown).
> >
> > Regards
> >
> > Antoine.
> 
> Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
> [GCC 4.4.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> x = intern("asdfqqqq")
> >>> id(x)
> 3078566112L
> >>> id("asdfqqqq")
> 3078566112L
> >>> x = []
> >>> y = []
> >>> id("asdfqqqq")
> 3078566208L
> 
> 
> Does not this suggest otherwise?

Oops. It looks like *I* have been mistaken, then. Sorry.

Regards

Antoine.




From tjreedy at udel.edu  Wed Apr 13 17:56:07 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 13 Apr 2011 11:56:07 -0400
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <1302704097.3565.1.camel@localhost.localdomain>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>	<BANLkTi=9iExR5aV6N7NHWzOj74qBB__8xw@mail.gmail.com>	<BANLkTimiBbJ_1BgePJNYKxagtyROaUtAAA@mail.gmail.com>	<io35uc$dk4$1@dough.gmane.org>
	<20110413135815.1fa4bbcc@pitrou.net>	<BANLkTikQQ2FAFTasZ9Wt_dH9w+gwHJHMyA@mail.gmail.com>
	<1302704097.3565.1.camel@localhost.localdomain>
Message-ID: <io4h2m$rjj$1@dough.gmane.org>

On 4/13/2011 10:14 AM, Antoine Pitrou wrote:
> Le mercredi 13 avril 2011 ? 09:47 -0400, Jim Jewett a ?crit :
>> On 4/13/11, Antoine Pitrou<solipsis at pitrou.net>  wrote:
>>> On Tue, 12 Apr 2011 23:40:02 -0400
>>> Terry Reedy<tjreedy at udel.edu>  wrote:
>>
>>>> "Changed in version 2.3: Interned strings are not immortal (like they
>>>> used to be in Python 2.2 and before); you must keep a reference to the
>>>> return value of intern() around to benefit from it."
>>
>>> That's a rather strange sentence, because interned strings *are*
>>> immortal (until the interpreter is shutdown).
>>
>> The purpose of that change (which may no longer be effective; I
>> haven't checked recently) was that they were no longer immortal.  If
>> the last reference outside the intern dictionary was removed, then the
>> string was removed from the intern dictionary as well.  Intern was a
>> way to de-duplicate, but it didn't (by itself) make anything immortal.
>
> They're de-facto immortal, since the user can't access the intern
> dictionary to remove these strings. That sentence looks like a very
> misleading way of explaining an implementation detail and making it look
> like a user-visible semantic change.

Quoted sentence was from 2.7. 3.2 has "Interned strings are not 
immortal; you must keep a reference to the return value of intern() 
around to benefit from it." This actually makes sense if true: if user 
cannot access string, it should go away. But I have no idea.

-- 
Terry Jan Reedy




From ericsnowcurrently at gmail.com  Wed Apr 13 18:22:36 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 13 Apr 2011 10:22:36 -0600
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTin5+T6qNNeZii8w0T93gH2=OsjyPg@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
	<BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
	<io1v0s$to3$1@dough.gmane.org>
	<BANLkTin5+T6qNNeZii8w0T93gH2=OsjyPg@mail.gmail.com>
Message-ID: <BANLkTimgw3XXwwXJsGwvvgKG2__8+2-v5g@mail.gmail.com>

On Tue, Apr 12, 2011 at 11:53 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> This whole discussion has been really useful to me in crystallising
> *why* I see value in PEP 3150, and it is directly related to the way
> function and class definitions work.
>
> I have the glimmerings of a rewrite of PEP 3150 kicking around in my
> skull, that may include restricting it to assignment statements (I'm
> not 100% decided on that point as yet - I'll make up my mind as the
> rest of the rewrite takes shape). The reason I am considering such a
> restriction is that the new Rationale section will likely be along the
> following lines:
>
> =========================
>
> Function and class statements in Python have a unique property
> relative to ordinary assignment statements: to some degree, they are
> *declarative*. They present the reader of the code with some critical
> information about a name that is about to be defined, before
> proceeding on with the details of the actual definition in the
> function or class body.
>
> The *name* of the object being declared is the first thing stated
> after the keyword. Other important information is also given the
> honour of preceding the implementation details:
>
> - decorators (which can greatly affect the behaviour of the created
> object, and were placed ahead of even the keyword and name as a matter
> of practicality moreso than aesthetics)
> - the docstring (on the first line immediately following the header line)
> - parameters, default values and annotations for function definitions
> - parent classes, metaclass and optionally other details (depending on
> the metaclass) for class definitions
>
> This PEP proposes to make a similar declarative style available for
> arbitrary assignment operations, by permitting the inclusion of a
> "given" suite following any simple (non-augmented) assignment
> statement::
>
>    TARGET = [TARGET2 = ... TARGETN =] EXPR given:
>        SUITE
>
> By convention, code in the body of the suite should be oriented solely
> towards correctly defining the assignment operation carried out in the
> header line. The header line operation should also be adequately
> descriptive (e.g. through appropriate choices of variable names) to
> give a reader a reasonable idea of the purpose of the operation
> without reading the body of the suite.
>
> =========================
>
> Another addition I am considering is the idea of allowing the "given"
> suite to contain a docstring, thus providing a way to make it easy to
> attach a __doc__ attribute to arbitrary targets. This may require
> disallowing tuple unpacking and multiple assignment targets when using
> the given clause, or else simply raising a syntax error if a docstring
> is present for an assignment using either of those forms.
>
> Other use cases will of course still be possible, but that will be the
> driving force behind the revised PEP.
>
>
Overall I like it.  The idea of limiting to assignment is a good one.  This
gives room for custom namespaces without all the class machinery.  It is
certainly something I have seen brought up on several occasions here.  And
these namespaces would not be anonymous since they are tied to the
assignment.  One benefit is that we could deprecate module variables using
this syntax.

-eric


> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110413/6f04e520/attachment.html>

From guido at python.org  Wed Apr 13 20:10:43 2011
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Apr 2011 11:10:43 -0700
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTin5+T6qNNeZii8w0T93gH2=OsjyPg@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
	<BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
	<io1v0s$to3$1@dough.gmane.org>
	<BANLkTin5+T6qNNeZii8w0T93gH2=OsjyPg@mail.gmail.com>
Message-ID: <BANLkTik_nHROwZ8+-eVodYdA-uuX-CCw8g@mail.gmail.com>

On Tue, Apr 12, 2011 at 10:53 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I have the glimmerings of a rewrite of PEP 3150 kicking around in my
> skull, that may include restricting it to assignment statements (I'm
> not 100% decided on that point as yet - I'll make up my mind as the
> rest of the rewrite takes shape). The reason I am considering such a
> restriction is that the new Rationale section will likely be along the
> following lines:
[...]

Hm. Most of the other simple statements currently mentioned in the PEP
make sense to me as well, e.g. "return X given: ...".

> Another addition I am considering is the idea of allowing the "given"
> suite to contain a docstring, thus providing a way to make it easy to
> attach a __doc__ attribute to arbitrary targets. This may require
> disallowing tuple unpacking and multiple assignment targets when using
> the given clause, or else simply raising a syntax error if a docstring
> is present for an assignment using either of those forms.

I like the idea of allowing a docstring in the given-clause, but I'm
not sure I care to enforce that the docstring is preserved somehow in
the target expression. And then again maybe if it's not preserved it's
not worth mentioning -- it can be considered a no-op just like putting
a "docstring" (really just a comment) in the middle of a block of
code, or e.g. at the top of a loop.

(All in all I think you have mostly managed to confuse me in this
message. At some point I even thought you meant that the body of the
given clause should be limited to definitions...)

-- 
--Guido van Rossum (python.org/~guido)


From greg.ewing at canterbury.ac.nz  Thu Apr 14 00:33:46 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 14 Apr 2011 10:33:46 +1200
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTin5+T6qNNeZii8w0T93gH2=OsjyPg@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
	<BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
	<io1v0s$to3$1@dough.gmane.org>
	<BANLkTin5+T6qNNeZii8w0T93gH2=OsjyPg@mail.gmail.com>
Message-ID: <4DA624CA.3070404@canterbury.ac.nz>

Nick Coghlan wrote:

> I have the glimmerings of a rewrite of PEP 3150 kicking around in my
> skull, that may include restricting it to assignment statements

Please don't, as that would eliminate a potentially useful
set of use cases.

Or at least it would be trying to, but it wouldn't be entirely
effective, because you would always be able to write

    dummy = something(foo) given:
      foo = ...

> Another addition I am considering is the idea of allowing the "given"
> suite to contain a docstring... This may require
> disallowing tuple unpacking and multiple assignment targets

I'm guessing you're thinking that the docstring would be
assigned to the __doc__ attribute of whatever object the
RHS expression returns.

I don't think there's any need to make a special case here
regarding unpacking. If it's not possible to assign a __doc__
to the object you'll get an exception, and if it works but
gets lost in the unpacking, too bad.

Also I can't see why multiple assignment targets would be a
problem at all -- the same object just gets assigned to all
the targets, including the __doc__.

-- 
Greg


From ncoghlan at gmail.com  Thu Apr 14 03:48:36 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Apr 2011 11:48:36 +1000
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <4DA624CA.3070404@canterbury.ac.nz>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<BANLkTimM2CcX+jBbkjiQYfbrctv188kPpg@mail.gmail.com>
	<BANLkTimCkZAgxjYhTXHMWWTN0F5TsO8BCw@mail.gmail.com>
	<BANLkTi=JzscZFGgSKt7=nGp=xXPi0hcCqA@mail.gmail.com>
	<BANLkTikv2AfvVJaHMQDEKbj5VdAU53wRzg@mail.gmail.com>
	<BANLkTi=n6dY18Bm3wdC204Jg08XhUj4Vqg@mail.gmail.com>
	<io1v0s$to3$1@dough.gmane.org>
	<BANLkTin5+T6qNNeZii8w0T93gH2=OsjyPg@mail.gmail.com>
	<4DA624CA.3070404@canterbury.ac.nz>
Message-ID: <BANLkTimeMmhzzCHyN184M9RWp-6=AYVDdA@mail.gmail.com>

On Thu, Apr 14, 2011 at 8:33 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>
>> I have the glimmerings of a rewrite of PEP 3150 kicking around in my
>> skull, that may include restricting it to assignment statements
>
> Please don't, as that would eliminate a potentially useful
> set of use cases.
>
> Or at least it would be trying to, but it wouldn't be entirely
> effective, because you would always be able to write
>
> ? dummy = something(foo) given:
> ? ? foo = ...

Yep, that's why I'm considering just describing the motivation in
terms of assignment, but then expanding it to other cases (including
"pass") to avoid silly workarounds when people decide to use it more
for the local namespace aspect than the "it's like 'def' for arbitrary
assignments" aspect (and "return" and "yield" have a lot in common
with assignment, anyway).

>> Another addition I am considering is the idea of allowing the "given"
>> suite to contain a docstring... This may require
>> disallowing tuple unpacking and multiple assignment targets
>
> I'm guessing you're thinking that the docstring would be
> assigned to the __doc__ attribute of whatever object the
> RHS expression returns.

Ah, true, that would make a lot of sense. It also generalises nicely
to other cases like "return" and "yield" as well. I'll see if I can
thrash out something sensible along those lines.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From cmjohnson.mailinglist at gmail.com  Thu Apr 14 07:46:45 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Wed, 13 Apr 2011 19:46:45 -1000
Subject: [Python-ideas] PEP-3150
Message-ID: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>

On Wed, Apr 13, 2011 at 3:48 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Thu, Apr 14, 2011 at 8:33 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

>> assigned to the __doc__ attribute of whatever object the
>> RHS expression returns.
>
> Ah, true, that would make a lot of sense. It also generalises nicely
> to other cases like "return" and "yield" as well. I'll see if I can
> thrash out something sensible along those lines.

So, in the case of

def decorator(f):
    return decorated given:
        "Docstring A..."
        def decorated(*args, **kwargs):
            "Docstring B..."
            do something...

it would be docstring A that gets put on the decorated function, not
docstring B? I guess I can kind of see the reasoning there, but it
seems a little weird.

Also, this case (and the general issue of callbacks, thunks, etc.)
seems to be crying out for a special convenience syntax to save a
level indenting. But would that just add needlessly to the complexity
of the language?

-- Carl


From ncoghlan at gmail.com  Thu Apr 14 08:03:03 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Apr 2011 16:03:03 +1000
Subject: [Python-ideas] PEP-3150
In-Reply-To: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>
References: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>
Message-ID: <BANLkTinq09WhuBb9jm_-EvMGObegwoAy_w@mail.gmail.com>

On Thu, Apr 14, 2011 at 3:46 PM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
> On Wed, Apr 13, 2011 at 3:48 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On Thu, Apr 14, 2011 at 8:33 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
>>> assigned to the __doc__ attribute of whatever object the
>>> RHS expression returns.
>>
>> Ah, true, that would make a lot of sense. It also generalises nicely
>> to other cases like "return" and "yield" as well. I'll see if I can
>> thrash out something sensible along those lines.
>
> So, in the case of
>
> def decorator(f):
> ? ?return decorated given:
> ? ? ? ?"Docstring A..."
> ? ? ? ?def decorated(*args, **kwargs):
> ? ? ? ? ? ?"Docstring B..."
> ? ? ? ? ? ?do something...
>
> it would be docstring A that gets put on the decorated function, not
> docstring B? I guess I can kind of see the reasoning there, but it
> seems a little weird.

Yep, but if it was designed that way from the start, people simply
wouldn't include the inner docstring.

I'm not completely sold on this particular idea as yet, but it's one
I'll explore further in the next update of the PEP.

> Also, this case (and the general issue of callbacks, thunks, etc.)
> seems to be crying out for a special convenience syntax to save a
> level indenting. But would that just add needlessly to the complexity
> of the language?

Given that PEP 3150 itself is already highly vulnerable to that last
accusation, I'm going to go with "Yes" :)

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From greg.ewing at canterbury.ac.nz  Thu Apr 14 10:06:36 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 14 Apr 2011 20:06:36 +1200
Subject: [Python-ideas] PEP-3150
In-Reply-To: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>
References: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>
Message-ID: <4DA6AB0C.3050403@canterbury.ac.nz>

Carl M. Johnson wrote:

> def decorator(f):
>     return decorated given:
>         "Docstring A..."
>         def decorated(*args, **kwargs):
>             "Docstring B..."
>             do something...
> 
> it would be docstring A that gets put on the decorated function, not
> docstring B?

That seems like a contrived example. If you wanted the
function to have Docstring B, why did you give the given
block a docstring of its own in the first place?

> Also, this case (and the general issue of callbacks, thunks, etc.)
> seems to be crying out for a special convenience syntax to save a
> level indenting.

In this particular case (i.e. a function that does nothing
but define and return another function) I'd like to be able
to write

   def decorator(f)(*args, **kwargs):
     do something...

but that's a subject for another hotly-debated PEP!

-- 
Greg


From cmjohnson.mailinglist at gmail.com  Thu Apr 14 13:47:32 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Thu, 14 Apr 2011 01:47:32 -1000
Subject: [Python-ideas] PEP-3150
In-Reply-To: <4DA6AB0C.3050403@canterbury.ac.nz>
References: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>
	<4DA6AB0C.3050403@canterbury.ac.nz>
Message-ID: <BANLkTim_PcRHYJSq7A=62J8Uv9Xu6QcC9w@mail.gmail.com>

On Wed, Apr 13, 2011 at 10:06 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:

> In this particular case (i.e. a function that does nothing
> but define and return another function) I'd like to be able
> to write
>
>  def decorator(f)(*args, **kwargs):
>    do something...
>
> but that's a subject for another hotly-debated PEP!

Interesting, but could that be extended to support making a throwaway
key function for sorted? With given it's obviously

sorted_list = sorted(original_list, key=keyfunc) given:
   def keyfunc(item):
       item = item.replace(" ", "")
       item = item.lowercase()
       ...
       return item

but I can't see how that your proposal could be expanded beyond the
one use case of decorator making, and today it's already possible to
make a simple decorator decorator that injects f as the first arg:

class MetaDec:
   def __init__(self, dec):
       self.dec = dec

   def __call__(self, f):
       def callme(*args, **kwargs):
            return self.dec(f, *args, **kwargs)
       return callme

@MetaDec
def logger(f, *args, **kwargs):
   print("Logging...")
   return f(*args, **kwargs)

@logger
def foo():
   print("Foo!")

I think the recipe for something like this is already in the docs
somewhere... Obviously, a built-in syntax for simple decorators might
have some savings in efficiency, but I would be surprised if it were
especially noteworthy, since decoration typically happens many fewer
times than function invocation.

Hmm, as I look at the given syntax here again, I find that I don't mind
the extra level of indention. Also, you can add another docstring to
clarify things a bit:


sorted_list = sorted(original_list, key=keyfunc) given:
   "A list of widgets sorted by removing whitespace and lowercasing..."
   def keyfunc(item):
       ...

>>> help(sorted_list)
"A list of widgets sorted by removing whitespace and lowercasing..."


From ncoghlan at gmail.com  Thu Apr 14 14:42:55 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Apr 2011 22:42:55 +1000
Subject: [Python-ideas] PEP-3150
In-Reply-To: <BANLkTim_PcRHYJSq7A=62J8Uv9Xu6QcC9w@mail.gmail.com>
References: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>
	<4DA6AB0C.3050403@canterbury.ac.nz>
	<BANLkTim_PcRHYJSq7A=62J8Uv9Xu6QcC9w@mail.gmail.com>
Message-ID: <BANLkTi=S-QKEMqBZNd-55x8HLxpYKwe=wg@mail.gmail.com>

On Thu, Apr 14, 2011 at 9:47 PM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
> sorted_list = sorted(original_list, key=keyfunc) given:
> ? "A list of widgets sorted by removing whitespace and lowercasing..."
> ? def keyfunc(item):
> ? ? ? ...
>
>>>> help(sorted_list)
> "A list of widgets sorted by removing whitespace and lowercasing..."

Alas, it isn't going to be quite that simple:

>>> x = []
>>> x.__doc__ = "This is a list"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object attribute '__doc__' is read-only
>>> x = 1
>>> x.__doc__ = "This is an integer"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object attribute '__doc__' is read-only

I don't see adding an extra pointer slot that will rarely be used to
the mutable builtins flying, and obviously the immutable types can't
reasonably accept a docstring that may vary by instance without
completely destroying their caching strategies.

Making the docstring available as a __doc__ variable in the given
namespace could potentially work, but I'm not sure the idea offers
much over an ordinary triple-quoted string variable at that point.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From arnodel at gmail.com  Thu Apr 14 14:57:45 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Thu, 14 Apr 2011 13:57:45 +0100
Subject: [Python-ideas] PEP-3150
In-Reply-To: <BANLkTim_PcRHYJSq7A=62J8Uv9Xu6QcC9w@mail.gmail.com>
References: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>
	<4DA6AB0C.3050403@canterbury.ac.nz>
	<BANLkTim_PcRHYJSq7A=62J8Uv9Xu6QcC9w@mail.gmail.com>
Message-ID: <43DD8F4A-3A70-4A0C-A8D5-1DB518CD2F37@gmail.com>


On 14 Apr 2011, at 12:47, Carl M. Johnson wrote:

> On Wed, Apr 13, 2011 at 10:06 PM, Greg Ewing
> <greg.ewing at canterbury.ac.nz> wrote:
> 
>> In this particular case (i.e. a function that does nothing
>> but define and return another function) I'd like to be able
>> to write
>> 
>> def decorator(f)(*args, **kwargs):
>>   do something...
>> 
>> but that's a subject for another hotly-debated PEP!
> 
> Interesting, but could that be extended to support making a throwaway
> key function for sorted? With given it's obviously
> 
> sorted_list = sorted(original_list, key=keyfunc) given:
>   def keyfunc(item):
>       item = item.replace(" ", "")
>       item = item.lowercase()
>       ...
>       return item
> 
> but I can't see how that your proposal could be expanded beyond the
> one use case of decorator making, and today it's already possible to
> make a simple decorator decorator that injects f as the first arg:
> 
> class MetaDec:
>   def __init__(self, dec):
>       self.dec = dec
> 
>   def __call__(self, f):
>       def callme(*args, **kwargs):
>            return self.dec(f, *args, **kwargs)
>       return callme

You could spell that:

from functools import partial
def MetaDec(f):
    return partial(partial, f)

And this is just the curry function!

-- 
Arnaud



From dag.odenhall at gmail.com  Thu Apr 14 17:00:13 2011
From: dag.odenhall at gmail.com (dag.odenhall at gmail.com)
Date: Thu, 14 Apr 2011 17:00:13 +0200
Subject: [Python-ideas] PEP-3150
In-Reply-To: <43DD8F4A-3A70-4A0C-A8D5-1DB518CD2F37@gmail.com>
References: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>
	<4DA6AB0C.3050403@canterbury.ac.nz>
	<BANLkTim_PcRHYJSq7A=62J8Uv9Xu6QcC9w@mail.gmail.com>
	<43DD8F4A-3A70-4A0C-A8D5-1DB518CD2F37@gmail.com>
Message-ID: <BANLkTin_oAQ-Rm_u6sO5uzkh-1pSStP3hQ@mail.gmail.com>

1. I don't like the special-cased docstring, but I suppose I could
simply opt to not use it myself.

2. I actually do like the added indentation; for me, a big point with
the given-statement is the readability, and the indentation hints at
the block's "secondary" nature and guides you toward the "primary"
statement.

One idea is to allow combining given and def:

    orderly = sorted(disorderly, key=getscore) given def getscore(item):
        return item.score

--
Dag


From arnodel at gmail.com  Thu Apr 14 18:34:11 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Thu, 14 Apr 2011 17:34:11 +0100
Subject: [Python-ideas] PEP-3150
In-Reply-To: <BANLkTin_oAQ-Rm_u6sO5uzkh-1pSStP3hQ@mail.gmail.com>
References: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>
	<4DA6AB0C.3050403@canterbury.ac.nz>
	<BANLkTim_PcRHYJSq7A=62J8Uv9Xu6QcC9w@mail.gmail.com>
	<43DD8F4A-3A70-4A0C-A8D5-1DB518CD2F37@gmail.com>
	<BANLkTin_oAQ-Rm_u6sO5uzkh-1pSStP3hQ@mail.gmail.com>
Message-ID: <11AE5D72-F492-4A3E-AFA5-579C3B7F4C9A@gmail.com>


On 14 Apr 2011, at 16:00, dag.odenhall at gmail.com wrote:

> 
>    orderly = sorted(disorderly, key=getscore) given def getscore(item):
>        return item.score

To me, this is not as readable as:

orderly = sorted(disorderly, key=getscore) given:
    def getscore(item):
        return item.score

Also, the related following is a good argument IMHO to allow expression statements before "given"

mylist.sort(key=getscore) given:
    def getscore(item):
        return item.score

If they were disallowed, I bet we would get "idioms" like:

_ = mylist.sort(key=getscore) given:
    def getscore(item):
        return item.score

-- 
Arnaud



From dag.odenhall at gmail.com  Thu Apr 14 18:53:15 2011
From: dag.odenhall at gmail.com (dag.odenhall at gmail.com)
Date: Thu, 14 Apr 2011 18:53:15 +0200
Subject: [Python-ideas] PEP-3150
In-Reply-To: <11AE5D72-F492-4A3E-AFA5-579C3B7F4C9A@gmail.com>
References: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>
	<4DA6AB0C.3050403@canterbury.ac.nz>
	<BANLkTim_PcRHYJSq7A=62J8Uv9Xu6QcC9w@mail.gmail.com>
	<43DD8F4A-3A70-4A0C-A8D5-1DB518CD2F37@gmail.com>
	<BANLkTin_oAQ-Rm_u6sO5uzkh-1pSStP3hQ@mail.gmail.com>
	<11AE5D72-F492-4A3E-AFA5-579C3B7F4C9A@gmail.com>
Message-ID: <BANLkTim=uWx_L64W94Tb1iKHyhx8cqbv9g@mail.gmail.com>

On 14 April 2011 18:34, Arnaud Delobelle <arnodel at gmail.com> wrote:
>
> On 14 Apr 2011, at 16:00, dag.odenhall at gmail.com wrote:
>
>>
>> ? ?orderly = sorted(disorderly, key=getscore) given def getscore(item):
>> ? ? ? ?return item.score
>
> To me, this is not as readable as:
>
> orderly = sorted(disorderly, key=getscore) given:
> ? ?def getscore(item):
> ? ? ? ?return item.score

I agree, I disliked the idea myself. There might be situations where
it'd be handy but I'm not convinced. If you need it because the code
is already indented several levels, it is likely that you won't fit
the "given def" on the same line anyway. If the function body is just
one statement you can inline it:

orderly = sorted(disorderly, key=getscore) given:
    def getscore(item): return item.score

If the function is larger and you're already deeply indented you
should probably refactor the code anyway.

>
> Also, the related following is a good argument IMHO to allow expression statements before "given"
>
> mylist.sort(key=getscore) given:
> ? ?def getscore(item):
> ? ? ? ?return item.score
>
> If they were disallowed, I bet we would get "idioms" like:
>
> _ = mylist.sort(key=getscore) given:
> ? ?def getscore(item):
> ? ? ? ?return item.score

Certainly agree - I didn't know it was proposed to limit it to assignment?


From solipsis at pitrou.net  Thu Apr 14 23:10:50 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 14 Apr 2011 23:10:50 +0200
Subject: [Python-ideas] Copy-on-write when forking a python process
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<BANLkTi=NDbSW96vVs7rwYrMFS9KfJu3_mA@mail.gmail.com>
Message-ID: <20110414231050.2f405afb@pitrou.net>

On Wed, 13 Apr 2011 16:17:03 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Wed, Apr 13, 2011 at 7:42 AM, jac <john.theman.connor-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org> wrote:
> > Has anyone else looked into the COW problem? ?Are there workarounds
> > and/or other plans to fix it? ?Does the solution I am proposing sound
> > reasonable, or does it seem like overkill? ?Does anyone see any
> > (technical) problems with it?
> 
> There's a clear workaround for the COW problem these days: use PyPy
> instead of CPython :)

Doesn't PyPy use a copying collector? Wouldn't that mean that the first
garbage collector pass after the fork() would un-share the memory pages
anyway?

Regards

Antoine.




From greg.ewing at canterbury.ac.nz  Fri Apr 15 02:50:16 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 15 Apr 2011 12:50:16 +1200
Subject: [Python-ideas] PEP-3150
In-Reply-To: <BANLkTin_oAQ-Rm_u6sO5uzkh-1pSStP3hQ@mail.gmail.com>
References: <BANLkTikK2N1qTPpzK5CJh1AEqYqOcNZ28w@mail.gmail.com>
	<4DA6AB0C.3050403@canterbury.ac.nz>
	<BANLkTim_PcRHYJSq7A=62J8Uv9Xu6QcC9w@mail.gmail.com>
	<43DD8F4A-3A70-4A0C-A8D5-1DB518CD2F37@gmail.com>
	<BANLkTin_oAQ-Rm_u6sO5uzkh-1pSStP3hQ@mail.gmail.com>
Message-ID: <4DA79648.70806@canterbury.ac.nz>

dag.odenhall at gmail.com wrote:

> One idea is to allow combining given and def:
> 
>     orderly = sorted(disorderly, key=getscore) given def getscore(item):
>         return item.score

I think I would actually prefer the 3-line version. There's
too much important stuff going on way over on the right there.

But here's another idea:

   orderly = sorted(disorderly, key = score) given:
     score(item) = item.score

-- 
Greg

> 
> --
> Dag
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



From stephen at xemacs.org  Fri Apr 15 07:33:30 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 15 Apr 2011 14:33:30 +0900
Subject: [Python-ideas] Assignments in list/generator expressions
In-Reply-To: <BANLkTimW5=4bMT4R19cc3GNiXArm3hz52A@mail.gmail.com>
References: <4D9FDF02.6080402@gmx.net>
	<BANLkTin1Wcyan-vY-19n+j-qyCcCq3VQ=g@mail.gmail.com>
	<BANLkTi=darjzwGpvwANdHAkKdktSxyPmKw@mail.gmail.com>
	<BANLkTikHzQUtvgdA5LLK2zu+FcZ2114RuQ@mail.gmail.com>
	<BANLkTikxti+XvVqn7qvaCK6ua5h2hN3MHQ@mail.gmail.com>
	<201104101513.p3AFDI94025509@theraft.openend.se>
	<BANLkTinOcy=gicVTbZBGvmhYTRV9MN82Rw@mail.gmail.com>
	<BANLkTikwUWPiQDWbfA+DFSDfiH1SzAg4Xw@mail.gmail.com>
	<eltoder@gmail.com>
	<BANLkTikF3Pc_oVwagYN3MCzLQXaiGFuh9A@mail.gmail.com>
	<201104110039.p3B0daqp011821@theraft.openend.se>
	<BANLkTikgqNGD7+jKf0u_SKX6vA31gn2cSQ@mail.gmail.com>
	<BANLkTinpUGd7-d39tVvGcnSDAEURhJaFAw@mail.gmail.com>
	<4DA38C08.7060308@canterbury.ac.nz>
	<BANLkTi=neOHFUjKD+KfLdCq1F74gQb4uog@mail.gmail.com>
	<4DA3996D.2000106@canterbury.ac.nz>
	<87bp0bzyno.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4DA4D262.3040805@canterbury.ac.nz>
	<878vvezy0z.fsf@uwakimon.sk.tsukuba.ac.jp>
	<BANLkTimW5=4bMT4R19cc3GNiXArm3hz52A@mail.gmail.com>
Message-ID: <8762qg84et.fsf@uwakimon.sk.tsukuba.ac.jp>

Jim Jewett writes:

 > I'm not sure there *is* a likable way to put the comprehension, the
 > temporary assignment, and the filter into one unbroken thought.

QOTW!



From ericsnowcurrently at gmail.com  Fri Apr 15 08:32:25 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 15 Apr 2011 00:32:25 -0600
Subject: [Python-ideas] PEP-3150
In-Reply-To: <BANLkTi=4V7r=vhk9cb8ckmfQbHaFSdfw1A@mail.gmail.com>
References: <BANLkTi=4V7r=vhk9cb8ckmfQbHaFSdfw1A@mail.gmail.com>
Message-ID: <BANLkTikQwqXJRamQdcYjT2OSePVfhbMs0g@mail.gmail.com>

I hadn't realized that the given syntax has been floated for years:

http://mail.python.org/pipermail/python-dev/2005-October/057409.html

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110415/11289615/attachment.html>

From ncoghlan at gmail.com  Fri Apr 15 09:24:54 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 15 Apr 2011 17:24:54 +1000
Subject: [Python-ideas] PEP-3150
In-Reply-To: <BANLkTikQwqXJRamQdcYjT2OSePVfhbMs0g@mail.gmail.com>
References: <BANLkTi=4V7r=vhk9cb8ckmfQbHaFSdfw1A@mail.gmail.com>
	<BANLkTikQwqXJRamQdcYjT2OSePVfhbMs0g@mail.gmail.com>
Message-ID: <BANLkTi=8tkMwzO86pKqdJbUTP2OawFOTMQ@mail.gmail.com>

On Fri, Apr 15, 2011 at 4:32 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> I hadn't realized that the given syntax has been floated for years:
> http://mail.python.org/pipermail/python-dev/2005-October/057409.html

Yep, that's the main reason I wrote PEP 3150 - so I didn't need to go
digging for old emails on the topic every time it came up :)

(with requests for multi-line lambdas being a common entry point into
the discussion!)

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From john.theman.connor at gmail.com  Fri Apr 15 16:17:41 2011
From: john.theman.connor at gmail.com (jac)
Date: Fri, 15 Apr 2011 07:17:41 -0700 (PDT)
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <20110413043411.028147f7@bhuda.mired.org>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<20110413043411.028147f7@bhuda.mired.org>
Message-ID: <744dee3c-4c79-42cd-8ed0-bacc0d91dcfd@t13g2000vbo.googlegroups.com>

> But maybe it's not premature. Do you have measurements that show how
> much extra swap space is taken up byCOWcopies caused by changing
> reference counts in your application?

In my case, memory almost the entire size of the dictionary is being
copied into the child process.  But for any specific case there will
be several factors involved:
The size of a page on the os
How many elements of the dictionary are accessed
The size of the objects in the dictionary (keys and values)
The distribution of the objects in memory. (For small objs you may
have more then one on a page, etc.)
etc.

But I think that it is possible in some cases that *more* memory then
the entire dictionary will be copied into the child process' memory.
I think that this would happen if each key value pair of the
dictionary were to be iterated over, and if the page size of the os
was larger then the size of an object, and the objects were arranged
in memory such that no two objects were contiguous.

--jac

On Apr 13, 3:34?am, Mike Meyer <m... at mired.org> wrote:
> On Tue, 12 Apr 2011 14:42:43 -0700 (PDT)
>
> jac <john.theman.con... at gmail.com> wrote:
> > Hi all,
> > Sorry for cross posting, but I think that this group may actually be
> > more appropriate for this discussion. ?Previous thread is at:
> >http://groups.google.com/group/comp.lang.python/browse_thread/thread/...
>
> > I am wondering if anything can be done about theCOW(copy-on-write)
> > problem when forking a python process. ?I have found several
> > discussions of this problem, but I have seen no proposed solutions or
> > workarounds. ?My understanding of the problem is that an object's
> > reference count is stored in the "ob_refcnt" field of the PyObject
> > structure itself. ?When a process forks, its memory is initially not
> > copied. However, if any references to an object are made or destroyed
> > in the child process, the page in which the objects "ob_refcnt" field
> > is located in will be copied.
>
> This smells like premature optimization to me. You're worried about
> the kernel copying a few extra pages of user data when you're dealing
> with a dictionary that's gigabytes in size. Sounds like any possibly
> memory savings here would be much smaller than those that could come
> from improving the data encoding.
>
> But maybe it's not premature. Do you have measurements that show how
> much extra swap space is taken up byCOWcopies caused by changing
> reference counts in your application?
>
> ? ? ? <mike
> --
> Mike Meyer <m... at mired.org> ? ? ? ? ? ? ?http://www.mired.org/consulting.html
> Independent Software developer/SCM consultant, email for more information.
>
> O< ascii ribbon campaign - stop html mail -www.asciiribbon.org
> _______________________________________________
> Python-ideas mailing list
> Python-id... at python.orghttp://mail.python.org/mailman/listinfo/python-ideas


From john.theman.connor at gmail.com  Fri Apr 15 16:25:42 2011
From: john.theman.connor at gmail.com (jac)
Date: Fri, 15 Apr 2011 07:25:42 -0700 (PDT)
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <BANLkTi=NDbSW96vVs7rwYrMFS9KfJu3_mA@mail.gmail.com>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<BANLkTi=NDbSW96vVs7rwYrMFS9KfJu3_mA@mail.gmail.com>
Message-ID: <049ee636-247b-4c45-95d9-d2a33954b4be@e8g2000vbz.googlegroups.com>

> There's a clear workaround for theCOWproblem these days: use PyPy
> instead of CPython :)

Thanks for the tip, I haven't looked at pypy in a while, it looks like
it has come a long way.  I will have to change some of my code around
to work with 2.5, but it shouldn't be too bad.  As far as I am
concerned, if pypy works for this, problem solved.

Thanks again,
--jac

On Apr 13, 1:17?am, Nick Coghlan <ncogh... at gmail.com> wrote:
> On Wed, Apr 13, 2011 at 7:42 AM, jac <john.theman.con... at gmail.com> wrote:
> > Has anyone else looked into theCOWproblem? ?Are there workarounds
> > and/or other plans to fix it? ?Does the solution I am proposing sound
> > reasonable, or does it seem like overkill? ?Does anyone see any
> > (technical) problems with it?
>
> There's a clear workaround for theCOWproblem these days: use PyPy
> instead of CPython :)
>
> Currently that workaround comes at a potentially high cost in
> compatibility with 3rd party C extensions, but that situation will
> naturally improve over time. Given that a lot of those compatibility
> problems arise *because* PyPy doesn't use refcounting natively, it's
> highly unlikely that there will be any significant tinkering with
> CPython's own approach.
>
> As far as technical problems go, opting out of memory management is a
> beautiful way to shoot yourself in the foot with memory leaks. All it
> takes is one optout() without a corresponding optin() and an arbitrary
> amount of memory may fail to be released. For example, in your own
> post, any exception in Fork_and_block_while_doing_stuff() means
> anything referenced directly or indirectly from mylist will be left
> hanging around in memory until the process terminates. That's a *far*
> worse problem than being unable to readily share memory between
> processes.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan?? |?? ncogh... at gmail.com?? |?? Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-id... at python.orghttp://mail.python.org/mailman/listinfo/python-ideas


From carl at oddbird.net  Fri Apr 15 18:42:18 2011
From: carl at oddbird.net (Carl Meyer)
Date: Fri, 15 Apr 2011 11:42:18 -0500
Subject: [Python-ideas] [Python-Dev] python and super
In-Reply-To: <4DA86C30.3010902@voidspace.org.uk>
References: <BANLkTimr4yWCCfgdm2KcWbpP-XcYP2SANw@mail.gmail.com>	<70EF2C52-1D92-4351-884E-52AF76BAAC6D@mac.com>	<4DA70ACA.4070204@voidspace.org.uk>	<20110414153503.F125B3A4063@sparrow.telecommunity.com>	<825E6CD5-8673-463B-92AE-59677C327C0A@gmail.com>	<4DA71C63.3030809@voidspace.org.uk>	<8A4A58EF-70F7-4F2F-8564-AE8611713986@mac.com>	<BANLkTincGnYrc48jzQfNrc_HFidohJXvwA@mail.gmail.com>	<4DA79E28.2060406@pearwood.info>	<4DA84DD6.20608@voidspace.org.uk>
	<4DA861B3.9010506@oddbird.net> <4DA86C30.3010902@voidspace.org.uk>
Message-ID: <4DA8756A.2040100@oddbird.net>

(Moving this thread to python-ideas, as it seems better suited for here).

On 04/15/2011 11:02 AM, Michael Foord wrote:
> Well yes, but it is also a bug in the copy of unittest2 embedded in
> django - so whilst it can be fixed in unittest2 (simply deleting the
> setUp and tearDown methods which do nothing but override
> unittest.TestCase.setUp and tearDown) it *also* needs to be fixed in
> django.

Yup.

> This particular issue does illustrate the problem well though - the
> methods in unittest2 don't call up to their parent class (which is fine
> because those methods are empty), but in not calling up also they
> prevent sibling methods being called in a multiple inheritance situation.
> 
> So for those who have been saying that not wanting to call up to parents
> is a valid use case, yes I quite agree.  But you have to be aware that
> because of the semantics of super, not calling up to your parents
> basically prevents those methods being used in the presence of multiple
> inheritance.

Right. I agree that the current behavior of super in that regard is
surprising, but I don't see how the "go ahead and call sibling methods
anyway" fix is plausible. What happens to the return value of that call?
There's nowhere for it to go - super() would have to somehow implicitly
integrate two different return values, which is all kinds of bad magic.
Consider these classes:

class UncooperativeBase:
    def method(self):
        # look Ma, no super() call!
        return ["UncooperativeBase"]


class OtherBase:
    def method(self):
        return ["OtherBase"] + super().method()


class Child(UncooperativeBase):
    def method(self):
        return ["Child"] + super().method()


class GrandChild(Child, OtherBase):
    def method(self):
        return ["GrandChild"] + super().method()


Currently, OtherBase.method() is never called, because
UncooperativeBase.method() breaks the super() chain. Any proposal to
have super() ensure that OtherBase.method() is called needs to explain
what exactly happens to its return value. This would normally be handled
explicitly by UncooperativeBase, since it would call super() and do
something with the return value. But with the explicit chain broken, you
end up with two different chains, and at some point their return values
would need integrating.

Carl


From ncoghlan at gmail.com  Fri Apr 15 18:56:38 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Apr 2011 02:56:38 +1000
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <049ee636-247b-4c45-95d9-d2a33954b4be@e8g2000vbz.googlegroups.com>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<BANLkTi=NDbSW96vVs7rwYrMFS9KfJu3_mA@mail.gmail.com>
	<049ee636-247b-4c45-95d9-d2a33954b4be@e8g2000vbz.googlegroups.com>
Message-ID: <BANLkTimfhqmueSK9MQE68_zo6pV4Zpud-Q@mail.gmail.com>

On Sat, Apr 16, 2011 at 12:25 AM, jac <john.theman.connor at gmail.com> wrote:
>> There's a clear workaround for theCOWproblem these days: use PyPy
>> instead of CPython :)
>
> Thanks for the tip, I haven't looked at pypy in a while, it looks like
> it has come a long way. ?I will have to change some of my code around
> to work with 2.5, but it shouldn't be too bad. ?As far as I am
> concerned, if pypy works for this, problem solved.

Their 2.7 compatible release should be out reasonably soon, too.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Fri Apr 15 19:04:26 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Apr 2011 03:04:26 +1000
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <20110414231050.2f405afb@pitrou.net>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<BANLkTi=NDbSW96vVs7rwYrMFS9KfJu3_mA@mail.gmail.com>
	<20110414231050.2f405afb@pitrou.net>
Message-ID: <BANLkTikkeORQsBFFBVrxfpjTj5_iQi5NMg@mail.gmail.com>

On Fri, Apr 15, 2011 at 7:10 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 13 Apr 2011 16:17:03 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> There's a clear workaround for the COW problem these days: use PyPy
>> instead of CPython :)
>
> Doesn't PyPy use a copying collector? Wouldn't that mean that the first
> garbage collector pass after the fork() would un-share the memory pages
> anyway?

I'm fairly sure one of the PyPy talks at Pycon specifically mentioned
the CoW problem as one of the ways PyPy was able to save memory over
CPython. The PyPy folks would be the ones to accurately answer
questions like that, though.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From mikegraham at gmail.com  Fri Apr 15 20:00:17 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Fri, 15 Apr 2011 14:00:17 -0400
Subject: [Python-ideas] [Python-Dev] python and super
In-Reply-To: <4DA8756A.2040100@oddbird.net>
References: <BANLkTimr4yWCCfgdm2KcWbpP-XcYP2SANw@mail.gmail.com>
	<70EF2C52-1D92-4351-884E-52AF76BAAC6D@mac.com>
	<4DA70ACA.4070204@voidspace.org.uk>
	<20110414153503.F125B3A4063@sparrow.telecommunity.com>
	<825E6CD5-8673-463B-92AE-59677C327C0A@gmail.com>
	<4DA71C63.3030809@voidspace.org.uk>
	<8A4A58EF-70F7-4F2F-8564-AE8611713986@mac.com>
	<BANLkTincGnYrc48jzQfNrc_HFidohJXvwA@mail.gmail.com>
	<4DA79E28.2060406@pearwood.info> <4DA84DD6.20608@voidspace.org.uk>
	<4DA861B3.9010506@oddbird.net> <4DA86C30.3010902@voidspace.org.uk>
	<4DA8756A.2040100@oddbird.net>
Message-ID: <BANLkTi=j-mehhsM+QY9CTndXsVR5vXhSvw@mail.gmail.com>

On Fri, Apr 15, 2011 at 12:42 PM, Carl Meyer <carl at oddbird.net> wrote:
> ...
> Consider these classes:
>
> class UncooperativeBase:
>    def method(self):
>        # look Ma, no super() call!
>        return ["UncooperativeBase"]
>
>
> class OtherBase:
>    def method(self):
>        return ["OtherBase"] + super().method()
>
>
> class Child(UncooperativeBase):
>    def method(self):
>        return ["Child"] + super().method()
>
>
> class GrandChild(Child, OtherBase):
>    def method(self):
>        return ["GrandChild"] + super().method()
>
>
> Currently, OtherBase.method() is never called, because
> UncooperativeBase.method() breaks the super() chain. Any proposal to
> have super() ensure that OtherBase.method() is called needs to explain
> what exactly happens to its return value. This would normally be handled
> explicitly by UncooperativeBase, since it would call super() and do
> something with the return value. But with the explicit chain broken, you
> end up with two different chains, and at some point their return values
> would need integrating.
>
> Carl

I feel like you're implying this would be fixed by having
UncooperativeBase call super().method(), but this would not make
working code. _It's the base class's responsibility *not* to call
super._

If UncooperativeBase called super() and we called
GrandChild().method(), then we would call our three superclasses's
method methods and object's. the problem is that object doesn't even
HAVE a method method, so we'll get an error. (For the very common
example of __init__, object *IS* the shared superless base class with
the method we want, but we need our own shared superless base class
for arbitrary methods.)

This isn't a problem I know a technical solution to?we can't have two
independent base classes with the same method. If they are related to
each other, they need to share a base class that probably doesn't do
anything exciting. If they're truly unrelated, then we don't want to
be calling them interchangeably in the MRO and we need to give them
different names, perhaps using a proxy class with one that calls into
the other.

Another solution that is much cleaner is to use composition for one or
both of the classes involved. This only works if these are concrete
classes, obviously, but gets us tons of benefits. It's not at all
complicated to decide what needs to be called when, change names, or
pick-and-choose what functionality to re-use.

Mike


From carl at oddbird.net  Fri Apr 15 20:17:11 2011
From: carl at oddbird.net (Carl Meyer)
Date: Fri, 15 Apr 2011 13:17:11 -0500
Subject: [Python-ideas] [Python-Dev] python and super
In-Reply-To: <BANLkTi=j-mehhsM+QY9CTndXsVR5vXhSvw@mail.gmail.com>
References: <BANLkTimr4yWCCfgdm2KcWbpP-XcYP2SANw@mail.gmail.com>	<70EF2C52-1D92-4351-884E-52AF76BAAC6D@mac.com>	<4DA70ACA.4070204@voidspace.org.uk>	<20110414153503.F125B3A4063@sparrow.telecommunity.com>	<825E6CD5-8673-463B-92AE-59677C327C0A@gmail.com>	<4DA71C63.3030809@voidspace.org.uk>	<8A4A58EF-70F7-4F2F-8564-AE8611713986@mac.com>	<BANLkTincGnYrc48jzQfNrc_HFidohJXvwA@mail.gmail.com>	<4DA79E28.2060406@pearwood.info>
	<4DA84DD6.20608@voidspace.org.uk>	<4DA861B3.9010506@oddbird.net>
	<4DA86C30.3010902@voidspace.org.uk>	<4DA8756A.2040100@oddbird.net>
	<BANLkTi=j-mehhsM+QY9CTndXsVR5vXhSvw@mail.gmail.com>
Message-ID: <4DA88BA7.8070104@oddbird.net>

Hi Mike,

On 04/15/2011 01:00 PM, Mike Graham wrote:
> On Fri, Apr 15, 2011 at 12:42 PM, Carl Meyer <carl at oddbird.net> wrote:
>> ...
>> Consider these classes:
>>
>> class UncooperativeBase:
>>    def method(self):
>>        # look Ma, no super() call!
>>        return ["UncooperativeBase"]
>>
>>
>> class OtherBase:
>>    def method(self):
>>        return ["OtherBase"] + super().method()
>>
>>
>> class Child(UncooperativeBase):
>>    def method(self):
>>        return ["Child"] + super().method()
>>
>>
>> class GrandChild(Child, OtherBase):
>>    def method(self):
>>        return ["GrandChild"] + super().method()
>>
>>
>> Currently, OtherBase.method() is never called, because
>> UncooperativeBase.method() breaks the super() chain. Any proposal to
>> have super() ensure that OtherBase.method() is called needs to explain
>> what exactly happens to its return value. This would normally be handled
>> explicitly by UncooperativeBase, since it would call super() and do
>> something with the return value. But with the explicit chain broken, you
>> end up with two different chains, and at some point their return values
>> would need integrating.
>>
>> Carl
> 
> I feel like you're implying this would be fixed by having
> UncooperativeBase call super().method(), but this would not make
> working code. _It's the base class's responsibility *not* to call
> super._
> 
> If UncooperativeBase called super() and we called
> GrandChild().method(), then we would call our three superclasses's
> method methods and object's. the problem is that object doesn't even
> HAVE a method method, so we'll get an error. (For the very common
> example of __init__, object *IS* the shared superless base class with
> the method we want, but we need our own shared superless base class
> for arbitrary methods.)

That's right, though only distantly related to my point. I was just too
aggressive in cutting the example down to a minimal number of classes;
UncooperativeBase and OtherBase should share a superclass with method(),
call it GrandParent.

My point still applies the same: if UncooperativeBase doesn't call
super().method(), it's not feasible for super to call OtherBase.method()
(the "call siblings anyway" feature that's been requested in this
thread), because it would introduce the need to somehow magically
"integrate" the return values from OtherBase.method() and
UncooperativeBase.method().

> This isn't a problem I know a technical solution to?we can't have two
> independent base classes with the same method. If they are related to
> each other, they need to share a base class that probably doesn't do
> anything exciting. If they're truly unrelated, then we don't want to
> be calling them interchangeably in the MRO and we need to give them
> different names, perhaps using a proxy class with one that calls into
> the other.

Right. This (unrelated base classes with the same method) was just an
error in my example, not a problem I am trying to propose a solution to.

> Another solution that is much cleaner is to use composition for one or
> both of the classes involved. This only works if these are concrete
> classes, obviously, but gets us tons of benefits. It's not at all
> complicated to decide what needs to be called when, change names, or
> pick-and-choose what functionality to re-use.

Agreed.

Carl


From greg.ewing at canterbury.ac.nz  Sat Apr 16 01:03:05 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 16 Apr 2011 11:03:05 +1200
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <20110413043411.028147f7@bhuda.mired.org>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<20110413043411.028147f7@bhuda.mired.org>
Message-ID: <4DA8CEA9.9000002@canterbury.ac.nz>

Mike Meyer wrote:
> You're worried about
> the kernel copying a few extra pages of user data when you're dealing
> with a dictionary that's gigabytes in size.

It's more than a few pages. If the entries are smaller than
a page, then looking at every entry will touch just about
every page.

-- 
Greg



From greg.ewing at canterbury.ac.nz  Sat Apr 16 02:03:48 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 16 Apr 2011 12:03:48 +1200
Subject: [Python-ideas] Copy-on-write when forking a python process
In-Reply-To: <BANLkTikkeORQsBFFBVrxfpjTj5_iQi5NMg@mail.gmail.com>
References: <cf8948a5-a2f8-4452-a9a6-82fbf40db313@p13g2000yqh.googlegroups.com>
	<BANLkTi=NDbSW96vVs7rwYrMFS9KfJu3_mA@mail.gmail.com>
	<20110414231050.2f405afb@pitrou.net>
	<BANLkTikkeORQsBFFBVrxfpjTj5_iQi5NMg@mail.gmail.com>
Message-ID: <4DA8DCE4.4050209@canterbury.ac.nz>

Nick Coghlan wrote:
> On Fri, Apr 15, 2011 at 7:10 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
>>Doesn't PyPy use a copying collector? Wouldn't that mean that the first
>>garbage collector pass after the fork() would un-share the memory pages
>>anyway?
> 
> I'm fairly sure one of the PyPy talks at Pycon specifically mentioned
> the CoW problem as one of the ways PyPy was able to save memory over
> CPython.

The answer is probably something along the lines that only the most
active parts of the heap get copied and the rest are left alone
most of the time. Otherwise copying GCs would be really bad for other
reasons too, such as causing a lot of paging and cache invalidation.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Sat Apr 16 02:12:21 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 16 Apr 2011 12:12:21 +1200
Subject: [Python-ideas] [Python-Dev] python and super
In-Reply-To: <BANLkTi=j-mehhsM+QY9CTndXsVR5vXhSvw@mail.gmail.com>
References: <BANLkTimr4yWCCfgdm2KcWbpP-XcYP2SANw@mail.gmail.com>
	<70EF2C52-1D92-4351-884E-52AF76BAAC6D@mac.com>
	<4DA70ACA.4070204@voidspace.org.uk>
	<20110414153503.F125B3A4063@sparrow.telecommunity.com>
	<825E6CD5-8673-463B-92AE-59677C327C0A@gmail.com>
	<4DA71C63.3030809@voidspace.org.uk>
	<8A4A58EF-70F7-4F2F-8564-AE8611713986@mac.com>
	<BANLkTincGnYrc48jzQfNrc_HFidohJXvwA@mail.gmail.com>
	<4DA79E28.2060406@pearwood.info> <4DA84DD6.20608@voidspace.org.uk>
	<4DA861B3.9010506@oddbird.net> <4DA86C30.3010902@voidspace.org.uk>
	<4DA8756A.2040100@oddbird.net>
	<BANLkTi=j-mehhsM+QY9CTndXsVR5vXhSvw@mail.gmail.com>
Message-ID: <4DA8DEE5.5070501@canterbury.ac.nz>

Mike Graham wrote:

> I feel like you're implying this would be fixed by having
> UncooperativeBase call super().method(), but this would not make
> working code. _It's the base class's responsibility *not* to call
> super._

I would say it's also the responsibility of every class
taking part in the chain to inherit directly or indirectly
from the common base class that defines the terminating
method, to ensure that it appears last in the mro.

-- 
Greg


From adam at matan.name  Sat Apr 16 13:50:55 2011
From: adam at matan.name (Adam Matan)
Date: Sat, 16 Apr 2011 14:50:55 +0300
Subject: [Python-ideas] Simple class initialization
Message-ID: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>

0. Abstract
===========

A class initialization often begins with a long list of explicit variable
declaration statements at the __init__() method. This repetitively copies
arguments into local data attributes.
This article suggests some semi-automatic techniques to shorten and clarify
this code section. Comments and responses are highly appreciated.

1. Credit
=========

The idea emerged from my question at stackoverflow.com. I would like to
thank
all those who answered and commented on that thread.

http://stackoverflow.com/questions/1389180

2. The problem
==============

Consider the following class:

        class Process:
            def __init__(self, pid, ppid, cmd, fd, reachable, user):

If the instance needs to hold these arguments internally, local data
attributes
are declared and the value of the argument is being copied, using two
mainstream
notations:

a.

                self.pid=pid
                self.ppid=ppid
                self.cmd=cmd
                self._fd=fd
                self.reachable=reachable
                self.user=user

b.

                self.pid, self.ppid, self.cmd, self._fd, self.reachable,
self.user = pid, ppid, cmd, fd, reachable, user

a. takes an unreasonable amount of lines and has a repetative form.
b. is long and prone to errors, especially when the argument list changes
   during development.

3. Solution outline
===================

1. Generally comply with the Zen of Python.
1. Explicit. The instance should not store any value unless told.
2. Short.
3. Backward compatible. Current Class syntax should work with the
   new solution.
4. Readable and intuitive.
5. Flexible. There should be a way to store any given subset of
   the arguments.
6. Allow storage of "private" variables by adding a single or double
   underscore before the variable name.

4. Solutions
============

4.1 Decorator
-------------

Nadia Alramli suggested this at the aforementiond thread at stackoverflow:

        from functools import wraps
        import inspect

        def initializer(fun):
            names, varargs, keywords, defaults = inspect.getargspec(fun)
            @wraps(fun)
            def wrapper(self, *args):
                for name, arg in zip(names[1:], args):
                    setattr(self, name, arg)
                fun(self, *args)
            return wrapper

        class Process:
            @initializer
            def __init__(self, pid, ppid, cmd, fd, reachable, user)

Pros:
    Simple, short, explicit and intuitive.
    Easy to add to the standard library, fully backward-compatible.

Cons:
    Stores all arguments.
    Does not support private data attributes notation (underscore prefix).

See
http://stackoverflow.com/questions/1389180/python-automatically-initialize-instance-variables/1389216#1389216


4.2. Argument tagging
---------------------

Arguments that needed to be stored within the instance could be marked with
a
special character, e.g. '~'. The character would be placed after the
argument
name for private variables:

        class Process:
            def __init__(self, ~pid, ~ppid, ~cmd, fd~, ~reachable, ~user)

Pros:
    Simple, short and explicit.
    Can store any subset of the arguments.
    Supports private variable notation.

Cons:
    Not intuitive. Changes the method signature and might be confusing.


4.3 Standard function
---------------------

A function will be called to store the argument as data attributes.

        class Process:
            def __init__(self, pid, ppid, cmd, fd, reachable, user)
                acquire(pid, ppid, cmd, reachable, user)
                acquire(fd, prefix='_')

Possible keywords can ba acquire, store, absorp.

Pros:
    Explicit, clear and intuitive.

Cons:
    Long - especially if more than a single prefix is used.

4.4 Initialization list
-----------------------

The argument list would include the name of the local data attribute, a
separator, and the argument name.

        class Process:
            def __init__(self, pid:pid, ppid:ppid, cmd:cmd, _fd:fd,
reachable:reachable, user:user)
            """ pid, ppid, cmd, reachable and user are stored as data
properties
                with the same name. fd is stored as _fd."""

Or:

        class Process:
            def __init__(self, :pid, :ppid, :cmd, _fd:fd, :reachable, :user)
            """Same, but local data attributes with the same name as
arguments
               would be stored without stating their name twice."""

This is a developed argument tagging (4.2).

Pros:
     See 4.2

Cons:
     Alters the method signature
     Not intuitive.


Looking forward for comments,
Adam matan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110416/92987df5/attachment.html>

From dag.odenhall at gmail.com  Sat Apr 16 14:08:45 2011
From: dag.odenhall at gmail.com (dag.odenhall at gmail.com)
Date: Sat, 16 Apr 2011 14:08:45 +0200
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
Message-ID: <BANLkTinvitTB0net598UgnQV9CBSO9rYLA@mail.gmail.com>

On 16 April 2011 13:50, Adam Matan <adam at matan.name> wrote:
> 0. Abstract
> ===========
> A class initialization often begins with a long list of explicit variable
> declaration statements at the __init__() method. This repetitively copies
> arguments into local data attributes.
> This article suggests some semi-automatic techniques to shorten and clarify
> this code section. Comments and responses are highly appreciated.
> 1. Credit
> =========
> The idea emerged from my question at stackoverflow.com. I would like to
> thank
> all those who answered and commented on that thread.
> http://stackoverflow.com/questions/1389180
> 2. The problem
> ==============
> Consider the following class:
> ? ? ? ? class Process:
> ? ? ? ? ? ? def __init__(self, pid, ppid, cmd, fd, reachable, user):
> If the instance needs to hold these arguments internally, local data
> attributes
> are declared and the value of the argument is being copied, using two
> mainstream
> notations:
> a.
> ? ? ? ? ? ? ? ? self.pid=pid
> ? ? ? ? ? ? ? ? self.ppid=ppid
> ? ? ? ? ? ? ? ? self.cmd=cmd
> ? ? ? ? ? ? ? ? self._fd=fd
> ? ? ? ? ? ? ? ? self.reachable=reachable
> ? ? ? ? ? ? ? ? self.user=user
> b.
> ? ? ? ? ? ? ? ? self.pid, self.ppid, self.cmd, self._fd, self.reachable,
> self.user = pid, ppid, cmd, fd, reachable, user
> a. takes an unreasonable amount of lines and has a repetative form.
> b. is long and prone to errors, especially when the argument list changes
> ? ?during development.
> 3. Solution outline
> ===================
> 1. Generally comply with the Zen of Python.
> 1. Explicit. The instance should not store any value unless told.
> 2. Short.
> 3. Backward compatible. Current Class syntax should work with the
> ? ?new solution.
> 4. Readable and intuitive.
> 5. Flexible. There should be a way to store any given subset of
> ? ?the arguments.
> 6. Allow storage of "private" variables by adding a single or double
> ? ?underscore before the variable name.
> 4. Solutions
> ============
> 4.1 Decorator
> -------------
> Nadia Alramli suggested this at the aforementiond thread at stackoverflow:
> ? ? ? ? from functools import wraps
> ? ? ? ? import inspect
> ? ? ? ? def initializer(fun):
> ? ? ? ? ? ? names, varargs, keywords, defaults = inspect.getargspec(fun)
> ? ? ? ? ? ? @wraps(fun)
> ? ? ? ? ? ? def wrapper(self, *args):
> ? ? ? ? ? ? ? ? for name, arg in zip(names[1:], args):
> ? ? ? ? ? ? ? ? ? ? setattr(self, name, arg)
> ? ? ? ? ? ? ? ? fun(self, *args)
> ? ? ? ? ? ? return wrapper
> ? ? ? ? class Process:
> ? ? ? ? ? ? @initializer
> ? ? ? ? ? ? def __init__(self, pid, ppid, cmd, fd, reachable, user)
> Pros:
> ? ? Simple, short, explicit and intuitive.
> ? ? Easy to add to the standard library, fully backward-compatible.
> Cons:
> ? ? Stores all arguments.
> ? ? Does not support private data attributes notation (underscore prefix).
> See
> http://stackoverflow.com/questions/1389180/python-automatically-initialize-instance-variables/1389216#1389216
>
> 4.2. Argument tagging
> ---------------------
> Arguments that needed to be stored within the instance could be marked with
> a
> special character, e.g. '~'. The character would be placed after the
> argument
> name for private variables:
> ? ? ? ? class Process:
> ? ? ? ? ? ? def __init__(self, ~pid, ~ppid, ~cmd, fd~, ~reachable, ~user)
> Pros:
> ? ? Simple, short and explicit.
> ? ? Can store any subset of the arguments.
> ? ? Supports private variable notation.
> Cons:
> ? ? Not intuitive. Changes the method signature and might be confusing.
>
> 4.3 Standard function
> ---------------------
> A function will be called to store the argument as data attributes.
> ? ? ? ? class Process:
> ? ? ? ? ? ? def __init__(self, pid, ppid, cmd, fd, reachable, user)
> ? ? ? ? ? ? ? ? acquire(pid, ppid, cmd, reachable, user)
> ? ? ? ? ? ? ? ? acquire(fd, prefix='_')
> Possible keywords can ba acquire, store, absorp.
> Pros:
> ? ? Explicit, clear and intuitive.
> Cons:
> ? ? Long - especially if more than a single prefix is used.
> 4.4 Initialization list
> -----------------------
> The argument list would include the name of the local data attribute, a
> separator, and the argument name.
> ? ? ? ? class Process:
> ? ? ? ? ? ? def __init__(self, pid:pid, ppid:ppid, cmd:cmd, _fd:fd,
> reachable:reachable, user:user)
> ? ? ? ? ? ? """ pid, ppid, cmd, reachable and user are stored as data
> properties
> ? ? ? ? ? ? ? ? with the same name. fd is stored as _fd."""
> Or:
> ? ? ? ? class Process:
> ? ? ? ? ? ? def __init__(self, :pid, :ppid, :cmd, _fd:fd, :reachable, :user)
> ? ? ? ? ? ? """Same, but local data attributes with the same name as
> arguments
> ? ? ? ? ? ? ? ?would be stored without stating their name twice."""
> This is a developed argument tagging (4.2).
> Pros:
> ? ? ?See 4.2
> Cons:
> ? ? ?Alters the method signature
> ? ? ?Not intuitive.
>
> Looking forward for comments,
> Adam matan


class Process:

    # Defaults and documentation as class attributes
    pid = None

    def __init__(self, **kwargs):
        for k, v in kwargs.iteritems():
            setattr(self, k, v)


From adam at matan.name  Sat Apr 16 14:14:00 2011
From: adam at matan.name (Adam Matan)
Date: Sat, 16 Apr 2011 15:14:00 +0300
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <BANLkTinvitTB0net598UgnQV9CBSO9rYLA@mail.gmail.com>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
	<BANLkTinvitTB0net598UgnQV9CBSO9rYLA@mail.gmail.com>
Message-ID: <BANLkTinyUDtC=-m9w4OrMPXtv=3qiGYJPg@mail.gmail.com>

I think that this solution damages the __init__() signature because the
caller does
not know which arguments should be passed. Furthermore, it is quite long,
and does
not allow introspection.


On Sat, Apr 16, 2011 at 3:08 PM, dag.odenhall at gmail.com <
dag.odenhall at gmail.com> wrote:

> On 16 April 2011 13:50, Adam Matan <adam at matan.name> wrote:
> > 0. Abstract
> > ===========
> > A class initialization often begins with a long list of explicit variable
> > declaration statements at the __init__() method. This repetitively copies
> > arguments into local data attributes.
> > This article suggests some semi-automatic techniques to shorten and
> clarify
> > this code section. Comments and responses are highly appreciated.
> > 1. Credit
> > =========
> > The idea emerged from my question at stackoverflow.com. I would like to
> > thank
> > all those who answered and commented on that thread.
> > http://stackoverflow.com/questions/1389180
> > 2. The problem
> > ==============
> > Consider the following class:
> >         class Process:
> >             def __init__(self, pid, ppid, cmd, fd, reachable, user):
> > If the instance needs to hold these arguments internally, local data
> > attributes
> > are declared and the value of the argument is being copied, using two
> > mainstream
> > notations:
> > a.
> >                 self.pid=pid
> >                 self.ppid=ppid
> >                 self.cmd=cmd
> >                 self._fd=fd
> >                 self.reachable=reachable
> >                 self.user=user
> > b.
> >                 self.pid, self.ppid, self.cmd, self._fd, self.reachable,
> > self.user = pid, ppid, cmd, fd, reachable, user
> > a. takes an unreasonable amount of lines and has a repetative form.
> > b. is long and prone to errors, especially when the argument list changes
> >    during development.
> > 3. Solution outline
> > ===================
> > 1. Generally comply with the Zen of Python.
> > 1. Explicit. The instance should not store any value unless told.
> > 2. Short.
> > 3. Backward compatible. Current Class syntax should work with the
> >    new solution.
> > 4. Readable and intuitive.
> > 5. Flexible. There should be a way to store any given subset of
> >    the arguments.
> > 6. Allow storage of "private" variables by adding a single or double
> >    underscore before the variable name.
> > 4. Solutions
> > ============
> > 4.1 Decorator
> > -------------
> > Nadia Alramli suggested this at the aforementiond thread at
> stackoverflow:
> >         from functools import wraps
> >         import inspect
> >         def initializer(fun):
> >             names, varargs, keywords, defaults = inspect.getargspec(fun)
> >             @wraps(fun)
> >             def wrapper(self, *args):
> >                 for name, arg in zip(names[1:], args):
> >                     setattr(self, name, arg)
> >                 fun(self, *args)
> >             return wrapper
> >         class Process:
> >             @initializer
> >             def __init__(self, pid, ppid, cmd, fd, reachable, user)
> > Pros:
> >     Simple, short, explicit and intuitive.
> >     Easy to add to the standard library, fully backward-compatible.
> > Cons:
> >     Stores all arguments.
> >     Does not support private data attributes notation (underscore
> prefix).
> > See
> >
> http://stackoverflow.com/questions/1389180/python-automatically-initialize-instance-variables/1389216#1389216
> >
> > 4.2. Argument tagging
> > ---------------------
> > Arguments that needed to be stored within the instance could be marked
> with
> > a
> > special character, e.g. '~'. The character would be placed after the
> > argument
> > name for private variables:
> >         class Process:
> >             def __init__(self, ~pid, ~ppid, ~cmd, fd~, ~reachable, ~user)
> > Pros:
> >     Simple, short and explicit.
> >     Can store any subset of the arguments.
> >     Supports private variable notation.
> > Cons:
> >     Not intuitive. Changes the method signature and might be confusing.
> >
> > 4.3 Standard function
> > ---------------------
> > A function will be called to store the argument as data attributes.
> >         class Process:
> >             def __init__(self, pid, ppid, cmd, fd, reachable, user)
> >                 acquire(pid, ppid, cmd, reachable, user)
> >                 acquire(fd, prefix='_')
> > Possible keywords can ba acquire, store, absorp.
> > Pros:
> >     Explicit, clear and intuitive.
> > Cons:
> >     Long - especially if more than a single prefix is used.
> > 4.4 Initialization list
> > -----------------------
> > The argument list would include the name of the local data attribute, a
> > separator, and the argument name.
> >         class Process:
> >             def __init__(self, pid:pid, ppid:ppid, cmd:cmd, _fd:fd,
> > reachable:reachable, user:user)
> >             """ pid, ppid, cmd, reachable and user are stored as data
> > properties
> >                 with the same name. fd is stored as _fd."""
> > Or:
> >         class Process:
> >             def __init__(self, :pid, :ppid, :cmd, _fd:fd, :reachable,
> :user)
> >             """Same, but local data attributes with the same name as
> > arguments
> >                would be stored without stating their name twice."""
> > This is a developed argument tagging (4.2).
> > Pros:
> >      See 4.2
> > Cons:
> >      Alters the method signature
> >      Not intuitive.
> >
> > Looking forward for comments,
> > Adam matan
>
>
> class Process:
>
>    # Defaults and documentation as class attributes
>    pid = None
>
>    def __init__(self, **kwargs):
>        for k, v in kwargs.iteritems():
>            setattr(self, k, v)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110416/48b22394/attachment.html>

From dag.odenhall at gmail.com  Sat Apr 16 14:48:18 2011
From: dag.odenhall at gmail.com (dag.odenhall at gmail.com)
Date: Sat, 16 Apr 2011 14:48:18 +0200
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <BANLkTinyUDtC=-m9w4OrMPXtv=3qiGYJPg@mail.gmail.com>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
	<BANLkTinvitTB0net598UgnQV9CBSO9rYLA@mail.gmail.com>
	<BANLkTinyUDtC=-m9w4OrMPXtv=3qiGYJPg@mail.gmail.com>
Message-ID: <BANLkTikUGVCM2-hjhpL6v+4ipYN78Eunbg@mail.gmail.com>

On 16 April 2011 14:14, Adam Matan <adam at matan.name> wrote:
> I think that this solution damages the __init__() signature because the
> caller does
> not know which arguments should be passed. Furthermore, it is quite long,
> and does
> not allow introspection.

That's why we have the class attributes, and document init as taking
kwargs that set the attributes.


From ethan at stoneleaf.us  Sat Apr 16 18:25:55 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 16 Apr 2011 09:25:55 -0700
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <BANLkTinvitTB0net598UgnQV9CBSO9rYLA@mail.gmail.com>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
	<BANLkTinvitTB0net598UgnQV9CBSO9rYLA@mail.gmail.com>
Message-ID: <4DA9C313.3060202@stoneleaf.us>

dag.odenhall at gmail.com wrote:
> On 16 April 2011 13:50, Adam Matan <adam at matan.name> wrote:
>> 0. Abstract
>> ===========
>> A class initialization often begins with a long list of explicit variable
>> declaration statements at the __init__() method. This repetitively copies
>> arguments into local data attributes.
>> This article suggests some semi-automatic techniques to shorten and clarify
>> this code section. Comments and responses are highly appreciated.

[snippers]

> class Process:
> 
>     # Defaults and documentation as class attributes
>     pid = None
> 
>     def __init__(self, **kwargs):
>         for k, v in kwargs.iteritems():
>             setattr(self, k, v)

I like the initialiser function myself:

8<-------------------------------------------------------------
def acquire(obj, kwargs):
     Missing = object()
     for kw, val in kwargs.items():
         name = '_'+kw
         attr = getattr(obj, name, Missing)
         if attr is Missing:
             name = kw
             attr = getattr(obj, name, Missing)
         if attr is not Missing:
             setattr(obj, name, val)

class Process:
     pid = None
     ppid = None
     cmd = None
     reachable = None
     user = None
     _fd = None
     def __init__(self, pid, ppid, cmd, fd, reachable, user):
         acquire(self, locals())
         print(self.pid)
         print(self.ppid)
         print(self.cmd)
         print(self.reachable)
         print(self.user)
         print(self._fd)

if __name__ == '__main__':
     p = Process(9, 101, 'cd /', 0, 'yes', 'root')
8<-------------------------------------------------------------

Don't think it needs to be in the stdlib, though.

~Ethan~


From fuzzyman at voidspace.org.uk  Sat Apr 16 19:22:44 2011
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 16 Apr 2011 18:22:44 +0100
Subject: [Python-ideas] [Python-Dev] python and super
In-Reply-To: <4DA8D6FC.9060707@canterbury.ac.nz>
References: <BANLkTimr4yWCCfgdm2KcWbpP-XcYP2SANw@mail.gmail.com>	<70EF2C52-1D92-4351-884E-52AF76BAAC6D@mac.com>	<4DA70ACA.4070204@voidspace.org.uk>	<20110414153503.F125B3A4063@sparrow.telecommunity.com>	<825E6CD5-8673-463B-92AE-59677C327C0A@gmail.com>	<4DA71C63.3030809@voidspace.org.uk>	<8A4A58EF-70F7-4F2F-8564-AE8611713986@mac.com>	<BANLkTincGnYrc48jzQfNrc_HFidohJXvwA@mail.gmail.com>	<4DA79E28.2060406@pearwood.info>
	<4DA84DD6.20608@voidspace.org.uk>
	<4DA8D6FC.9060707@canterbury.ac.nz>
Message-ID: <4DA9D064.3050904@voidspace.org.uk>

On 16/04/2011 00:38, Greg Ewing wrote:
> Michael Foord wrote:
>
>> consider the "recently" introduced problem caused by object.__init__
> > not taking arguments. This makes it impossible to use super correctly
> > in various circumstances.
> >
> > ...
> >
>> It is impossible to inherit from both C and A and have all parent
>> __init__ methods called correctly. Changing the semantics of super as
>> described would fix this problem.
>
> I don't see how, because auto-super-calling would eventually
> end up trying to call object.__init__ with arguments and fail.
>

No, not as I described. Where a method does not call up to its parent 
class then super would *not* auto call the parent class (I'm *not* 
suggesting a fully auto-call super as the original poster did), but a 
method not calling super still wouldn't halt the chain of calls. To 
achieve this a call into a method that doesn't call super (and would 
normally end the chain) instead removes itself and its unique parents 
[1] from the mro for this call.

It would make the semantics more complex, but it would solve this 
problem and the original posters problem. For this specific example, if 
no classes call up to object.__init__ then it won't be called. It would 
permit you to have methods that can participate in multiple inheritance 
whilst still blocking calls (overriding) up to their parent classes.

All the best,

Michael

[1] i.e. classes that aren't also the parent of another class still in 
the mro.

> You might think to "fix" this by making a special case of
> object.__init__ and refraining from calling it. But the same
> problem arises in a more general way whenever some class in
> the mix has a method with the right name but the wrong
> signature, which is likely to happen if you try to mix
> classes that weren't designed to be mixed together.
>


-- 
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html



From nathan at cmu.edu  Sat Apr 16 19:23:13 2011
From: nathan at cmu.edu (Nathan Schneider)
Date: Sat, 16 Apr 2011 13:23:13 -0400
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <4DA9C313.3060202@stoneleaf.us>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
	<BANLkTinvitTB0net598UgnQV9CBSO9rYLA@mail.gmail.com>
	<4DA9C313.3060202@stoneleaf.us>
Message-ID: <BANLkTing=wM2UNfeN9vz986cotyJkztpgw@mail.gmail.com>

I like the initializer decorator. Here's a version that (1) handles keyword
argument defaults, and (2) allows only a subset of arguments to be stored
via decorator arguments:

def initializer(*selectedArgs):
    def wrap(fun):
        names, varargs, varkwargs, defaults = inspect.getargspec(fun)
        @wraps(fun)
        def wrapper(self, *args, **kwargs):
            d = dict(zip(names[-len(defaults):],defaults))
            d.update(dict(zip(names[1:], args)))
            d.update(kwargs)
            for a in (selectedArgs if len(selectedArgs)>0 else d.keys()):
                assert a in names,'Invalid parameter name: {}'.format(a)
                assert a in d,'Missing required argument: {}'.format(a)
                setattr(self, a, d[a])
            fun(self, *args, **kwargs)
        return wrapper
    return wrap

class Process1:
    @initializer()
    def __init__(self, pid, ppid, cmd, fd, reachable=True, user=None)

class Process2:
    @initializer('pid','ppid','user') # only store these 3; self.cmd will
trigger an error
    def __init__(self, pid, ppid, cmd, fd, reachable=True, user=None)

Nathan

On Sat, Apr 16, 2011 at 12:25 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
> dag.odenhall at gmail.com wrote:
>>
>> On 16 April 2011 13:50, Adam Matan <adam at matan.name> wrote:
>>>
>>> 0. Abstract
>>> ===========
>>> A class initialization often begins with a long list of explicit
variable
>>> declaration statements at the __init__() method. This repetitively
copies
>>> arguments into local data attributes.
>>> This article suggests some semi-automatic techniques to shorten and
>>> clarify
>>> this code section. Comments and responses are highly appreciated.
>
> [snippers]
>
>> class Process:
>>
>>    # Defaults and documentation as class attributes
>>    pid = None
>>
>>    def __init__(self, **kwargs):
>>        for k, v in kwargs.iteritems():
>>            setattr(self, k, v)
>
> I like the initialiser function myself:
>
> 8<-------------------------------------------------------------
> def acquire(obj, kwargs):
>    Missing = object()
>    for kw, val in kwargs.items():
>        name = '_'+kw
>        attr = getattr(obj, name, Missing)
>        if attr is Missing:
>            name = kw
>            attr = getattr(obj, name, Missing)
>        if attr is not Missing:
>            setattr(obj, name, val)
>
> class Process:
>    pid = None
>    ppid = None
>    cmd = None
>    reachable = None
>    user = None
>    _fd = None
>    def __init__(self, pid, ppid, cmd, fd, reachable, user):
>        acquire(self, locals())
>        print(self.pid)
>        print(self.ppid)
>        print(self.cmd)
>        print(self.reachable)
>        print(self.user)
>        print(self._fd)
>
> if __name__ == '__main__':
>    p = Process(9, 101, 'cd /', 0, 'yes', 'root')
> 8<-------------------------------------------------------------
>
> Don't think it needs to be in the stdlib, though.
>
> ~Ethan~
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110416/9c1cd101/attachment.html>

From arnodel at gmail.com  Sat Apr 16 21:58:41 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Sat, 16 Apr 2011 20:58:41 +0100
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
Message-ID: <89CDC541-02EE-42BE-A48E-F02EC078B886@gmail.com>


On 16 Apr 2011, at 12:50, Adam Matan wrote:

> 0. Abstract
> ===========
> 
> A class initialization often begins with a long list of explicit variable 
> declaration statements at the __init__() method. This repetitively copies 
> arguments into local data attributes.
> This article suggests some semi-automatic techniques to shorten and clarify 
> this code section. Comments and responses are highly appreciated.

Following a discussion on c.l.python, I posted a recipe on ActiveState a while ago that attempted to deal with this issue:

    http://code.activestate.com/recipes/551763-automatic-attribute-assignment/

From the docstring:

    """
    autoassign(function) -> method
    autoassign(*argnames) -> decorator
    autoassign(exclude=argnames) -> decorator
    
    allow a method to assign (some of) its arguments as attributes of
    'self' automatically.  E.g.
    
    >>> class Foo(object):
    ...     @autoassign
    ...     def __init__(self, foo, bar): pass
    ... 
    >>> breakfast = Foo('spam', 'eggs')
    >>> breakfast.foo, breakfast.bar
    ('spam', 'eggs')
    
    To restrict autoassignment to 'bar' and 'baz', write:
    
        @autoassign('bar', 'baz')
        def method(self, foo, bar, baz): ...

    To prevent 'foo' and 'baz' from being autoassigned, use:

        @autoassign(exclude=('foo', 'baz'))
        def method(self, foo, bar, baz): ...
    """

-- 
Arnaud



From adam at matan.name  Sat Apr 16 23:10:29 2011
From: adam at matan.name (Adam Matan)
Date: Sun, 17 Apr 2011 00:10:29 +0300
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <89CDC541-02EE-42BE-A48E-F02EC078B886@gmail.com>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
	<89CDC541-02EE-42BE-A48E-F02EC078B886@gmail.com>
Message-ID: <BANLkTikGZoD+RVNrsCUUeiASL=pyz0ZF=A@mail.gmail.com>

I think autoassign is a suitable name for this functionality.
I would like to see this as a PEP.
Adam

On Sat, Apr 16, 2011 at 10:58 PM, Arnaud Delobelle <arnodel at gmail.com>wrote:

>
> On 16 Apr 2011, at 12:50, Adam Matan wrote:
>
> > 0. Abstract
> > ===========
> >
> > A class initialization often begins with a long list of explicit variable
> > declaration statements at the __init__() method. This repetitively copies
> > arguments into local data attributes.
> > This article suggests some semi-automatic techniques to shorten and
> clarify
> > this code section. Comments and responses are highly appreciated.
>
> Following a discussion on c.l.python, I posted a recipe on ActiveState a
> while ago that attempted to deal with this issue:
>
>
> http://code.activestate.com/recipes/551763-automatic-attribute-assignment/
>
> From the docstring:
>
>    """
>    autoassign(function) -> method
>    autoassign(*argnames) -> decorator
>    autoassign(exclude=argnames) -> decorator
>
>    allow a method to assign (some of) its arguments as attributes of
>    'self' automatically.  E.g.
>
>    >>> class Foo(object):
>    ...     @autoassign
>    ...     def __init__(self, foo, bar): pass
>    ...
>    >>> breakfast = Foo('spam', 'eggs')
>    >>> breakfast.foo, breakfast.bar
>    ('spam', 'eggs')
>
>    To restrict autoassignment to 'bar' and 'baz', write:
>
>        @autoassign('bar', 'baz')
>        def method(self, foo, bar, baz): ...
>
>    To prevent 'foo' and 'baz' from being autoassigned, use:
>
>        @autoassign(exclude=('foo', 'baz'))
>        def method(self, foo, bar, baz): ...
>    """
>
> --
> Arnaud
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110417/87186e15/attachment.html>

From pyideas at rebertia.com  Sun Apr 17 00:53:50 2011
From: pyideas at rebertia.com (Chris Rebert)
Date: Sat, 16 Apr 2011 15:53:50 -0700
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
Message-ID: <BANLkTi=QbGDU1NXX4wYsAXvB57Ztwn4xFA@mail.gmail.com>

On Sat, Apr 16, 2011 at 4:50 AM, Adam Matan <adam at matan.name> wrote:
> 0. Abstract
> ===========
> A class initialization often begins with a long list of explicit variable
> declaration statements at the __init__() method. This repetitively copies
> arguments into local data attributes.
> This article suggests some semi-automatic techniques to shorten and clarify
> this code section. Comments and responses are highly appreciated.
<snip>
> 4. Solutions
> ============
> 4.1 Decorator
> -------------
<snip>
>         class Process:
>             @initializer
>             def __init__(self, pid, ppid, cmd, fd, reachable, user):
<snip>
> 4.2. Argument tagging
> ---------------------
> Arguments that needed to be stored within the instance could be marked with
> a
> special character, e.g. '~'. The character would be placed after the
> argument
> name for private variables:
> ? ? ? ? class Process:
> ? ? ? ? ? ? def __init__(self, ~pid, ~ppid, ~cmd, fd~, ~reachable, ~user)
> Pros:
> ? ? Simple, short and explicit.
> ? ? Can store any subset of the arguments.
> ? ? Supports private variable notation.
> Cons:
> ? ? Not intuitive. Changes the method signature and might be confusing.

You could easily combine 4.1 and 4.2 by using function annotations
(PEP 3107 - http://www.python.org/dev/peps/pep-3107/ ); this would
eliminate the need to add any new syntax. Example:

from wherever import initializer, Private as Priv, Public as Pub
class Process:
    @initializer
    def __init__(self, pid: Pub, ppid: Pub, cmd: Pub, fd: Priv,
reachable: Pub, user: Pub):

> 4.3 Standard function
> ---------------------
> A function will be called to store the argument as data attributes.
> ? ? ? ? class Process:
> ? ? ? ? ? ? def __init__(self, pid, ppid, cmd, fd, reachable, user)
> ? ? ? ? ? ? ? ? acquire(pid, ppid, cmd, reachable, user)
> ? ? ? ? ? ? ? ? acquire(fd, prefix='_')
> Possible keywords can ba acquire, store, absorp.
> Pros:
> ? ? Explicit, clear and intuitive.

-1; I strongly disagree. This function would have to be rather magical
since `self` isn't passed to it; it would have to mess with call stack
frames to grab `self` from the caller's scope. I'm unsure whether that
would be resilient in the face of methods which name `self` something
else (e.g. `s`), and whether that would be easily portable to
non-CPython implementations.

> Cons:
> ? ? Long - especially if more than a single prefix is used.
> 4.4 Initialization list
> -----------------------
> The argument list would include the name of the local data attribute, a
> separator, and the argument name.
> ? ? ? ? class Process:
> ? ? ? ? ? ? def __init__(self, pid:pid, ppid:ppid, cmd:cmd, _fd:fd,
> reachable:reachable, user:user)
> ? ? ? ? ? ? """ pid, ppid, cmd, reachable and user are stored as data
> properties
> ? ? ? ? ? ? ? ? with the same name. fd is stored as _fd."""
> Or:
> ? ? ? ? class Process:
> ? ? ? ? ? ? def __init__(self, :pid, :ppid, :cmd, _fd:fd, :reachable, :user)
> ? ? ? ? ? ? """Same, but local data attributes with the same name as
> arguments
> ? ? ? ? ? ? ? ?would be stored without stating their name twice."""
> This is a developed argument tagging (4.2).

Again, I think function annotations would be a better approach here. Example:

class Process:
    @initializer
    def __init__(self, pid: 'pid', ppid: 'ppid', cmd: 'cmd', fd:
'_fd', reachable: 'reachable', user: 'user'):

or allowing for more implicitness:

class Process:
    @initializer
    def __init__(self, pid, ppid, cmd, fd: '_fd', reachable, user):


Cheers,
Chris
--
http://blog.rebertia.com


From bruce at leapyear.org  Sun Apr 17 02:46:42 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Sat, 16 Apr 2011 17:46:42 -0700
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
Message-ID: <BANLkTimhHupAzHWSCxQt+cYpvLpOoPECkw@mail.gmail.com>

For:

def __init__(self, pid:pid, ppid:ppid, cmd:cmd, _fd:fd, reachable:reachable,
user:user)


This either conflicts with parameter annotations or you've got the
annotation on the wrong side (and annotations are expressions so this won't
work). I had a similar idea to what Chris Rebert suggested:

from somewhere import auto_init
@auto_init
def __init__(self, pid, ppid, cmd, fd:[auto_init.private], reachable:[
auto_init.skip], user:[auto_init.name('user_name')])
   blah


The annotation auto_init.private is equivalent to auto_init.name('_'+*
parameter_name*).

Note that I wrote fd:[auto_init.private] instead of auto_init.private. One
of the strange aspects (to me) of parameter annotations is that they have no
semantics which opens them up to multiple conflicting uses. If we
standardize on a convention that the annotation is a list (or tuple)
of annotations, then this leads us to usage like

foo:[auto_init.name('bar'),constraint.non_negative,etc].


--- Bruce
*New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April
Fools!
*New!** *Blog post: http://www.vroospeak.com Ironically, a glaring Google
grammatical error
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110416/a18d9b29/attachment.html>

From steve at pearwood.info  Sun Apr 17 03:27:32 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 17 Apr 2011 11:27:32 +1000
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <BANLkTimhHupAzHWSCxQt+cYpvLpOoPECkw@mail.gmail.com>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
	<BANLkTimhHupAzHWSCxQt+cYpvLpOoPECkw@mail.gmail.com>
Message-ID: <4DAA4204.7020407@pearwood.info>

Bruce Leban wrote:

> One
> of the strange aspects (to me) of parameter annotations is that they have no
> semantics which opens them up to multiple conflicting uses.

That's not a bug, that's a feature.

It's been stated many times by Guido that it's far too early to 
standardize on a single meaning for annotations. (We may *never* 
standardize on a single meaning.) Instead, it is up to the library or 
decorator to impose whatever meaning makes sense for that particular 
library or decorator.



-- 
Steven



From bruce at leapyear.org  Sun Apr 17 06:53:18 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Sat, 16 Apr 2011 21:53:18 -0700
Subject: [Python-ideas] parameter annotation semantics
Message-ID: <BANLkTikMe2tt-0yd77RQgqdk0BvL+MwpGw@mail.gmail.com>

Subject was: Re: [Python-ideas] Simple class initialization



On Sat, Apr 16, 2011 at 6:27 PM, Steven D'Aprano <steve at pearwood.info>wrote:

> Bruce Leban wrote:
>
>  One
>> of the strange aspects (to me) of parameter annotations is that they have
>> no
>> semantics which opens them up to multiple conflicting uses.
>>
>
> That's not a bug, that's a feature.
>
> It's been stated many times by Guido that it's far too early to standardize
> on a single meaning for annotations. (We may *never* standardize on a single
> meaning.) Instead, it is up to the library or decorator to impose whatever
> meaning makes sense for that particular library or decorator.
>


I understand that. I can still think it a bit strange, can't I? To be more
specific, you hit the crux of the problem with the statement "it is up to *
THE* library or decorator to impose whatever meaning makes sense" [emphasis
added] is that it assumes the singular. If I want to use two decorators
which impose different meanings, I'm stuck. Imagine I have two decorators
like this:

@memoize  # memoizes return values; if memo_value annotation is used,
memoizes using
          # the value returned by applying the function
def a(foo:memo_value(int), bar:memo_value(tuple)):
  pass

@log_me  # logs function calls and return values excluding parameters
annotated no_log
def b(foo, bar:no_log):
  pass


Can I use both @memoize and @logging? Not if they're defined as they are
above. I can if the decorators expect a list of annotations instead of a
solo annotation:

@memoize
def a(foo:[memo_value(int)], bar:[memo_value(tuple)]):
  pass

@log_me
def b(foo, bar:[no_log]):
  pass

@memoize
@log_me
def c(foo:[memo_value(int)], bar:[no_log, memo_value(tuple)]):
  pass


Note that I am NOT proposing a change to the language (none is necessary),
just suggesting that this would be good advice to offer authors. PEP 8
offers no advice on annotation syntax and PEP 3107 explicitly does not offer
any advice saying "Despite yet more discussion, it was decided not to
standardize a mechanism for annotation interoperability."

What I would propose is something more like this:

There is no standard for annotation interoperability. Annotations that wish
to support interoperability should consider treating the annotation as an
enumerable of values and checking value in annotation  (or something
similar) instead of value == annotation.


Of course an obvious alternative is

@memoize
@log_me
def c(foo:{memoize:memo_value(int)}, bar:{log_me:no_log,
memoize:memo_value(tuple)}):
  pass


but I think that's too verbose and therefore less likely to be adopted.

--- Bruce
*New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April
Fools!
*New!** *Blog post: http://www.vroospeak.com Ironically, a glaring Google
grammatical error
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110416/3d097a93/attachment.html>

From stefan_ml at behnel.de  Sun Apr 17 08:26:13 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 17 Apr 2011 08:26:13 +0200
Subject: [Python-ideas] parameter annotation semantics
In-Reply-To: <BANLkTikMe2tt-0yd77RQgqdk0BvL+MwpGw@mail.gmail.com>
References: <BANLkTikMe2tt-0yd77RQgqdk0BvL+MwpGw@mail.gmail.com>
Message-ID: <ioe165$eho$1@dough.gmane.org>

Bruce Leban, 17.04.2011 06:53:
> Subject was: Re: [Python-ideas] Simple class initialization
> On Sat, Apr 16, 2011 at 6:27 PM, Steven D'Aprano wrote:
>> Bruce Leban wrote:
>>> One
>>> of the strange aspects (to me) of parameter annotations is that they have
>>> no semantics which opens them up to multiple conflicting uses.
>>
>> That's not a bug, that's a feature.
>>
>> It's been stated many times by Guido that it's far too early to standardize
>> on a single meaning for annotations. (We may *never* standardize on a single
>> meaning.) Instead, it is up to the library or decorator to impose whatever
>> meaning makes sense for that particular library or decorator.
>
> I understand that. I can still think it a bit strange, can't I? To be more
> specific, you hit the crux of the problem with the statement "it is up to *
> THE* library or decorator to impose whatever meaning makes sense" [emphasis
> added] is that it assumes the singular. If I want to use two decorators
> which impose different meanings, I'm stuck. Imagine I have two decorators
> like this:
>
> @memoize  # memoizes return values; if memo_value annotation is used,
> memoizes using
>            # the value returned by applying the function
> def a(foo:memo_value(int), bar:memo_value(tuple)):
>    pass
>
> @log_me  # logs function calls and return values excluding parameters
> annotated no_log
> def b(foo, bar:no_log):
>    pass
>
> Can I use both @memoize and @logging?

FWIW, the implementation in Cython accepts tuples as annotation values and 
extracts relevant type annotations from suitable values in them. That was 
chosen because a tuple appeared to be the only reasonable format at the time.

Stefan



From pyideas at rebertia.com  Sun Apr 17 08:36:34 2011
From: pyideas at rebertia.com (Chris Rebert)
Date: Sat, 16 Apr 2011 23:36:34 -0700
Subject: [Python-ideas] parameter annotation semantics
In-Reply-To: <BANLkTikMe2tt-0yd77RQgqdk0BvL+MwpGw@mail.gmail.com>
References: <BANLkTikMe2tt-0yd77RQgqdk0BvL+MwpGw@mail.gmail.com>
Message-ID: <BANLkTincMyv39L0KyvVnPvj5iPkPpf_OYA@mail.gmail.com>

On Sat, Apr 16, 2011 at 9:53 PM, Bruce Leban <bruce at leapyear.org> wrote:
> Subject was:?Re: [Python-ideas] Simple class initialization
> On Sat, Apr 16, 2011 at 6:27 PM, Steven D'Aprano <steve at pearwood.info>
> wrote:
>> Bruce Leban wrote:
>>> One
>>> of the strange aspects (to me) of parameter annotations is that they have
>>> no
>>> semantics which opens them up to multiple conflicting uses.
>>
>> That's not a bug, that's a feature.
>>
>> It's been stated many times by Guido that it's far too early to
>> standardize on a single meaning for annotations. (We may *never* standardize
>> on a single meaning.) Instead, it is up to the library or decorator to
>> impose whatever meaning makes sense for that particular library or
>> decorator.
>
> I understand that. I can still think it a bit strange, can't I? To be more
> specific, you hit the crux of the problem with the statement "it is up to
> THE library or decorator to impose whatever meaning makes sense" [emphasis
> added] is that it assumes the singular. If I want to use two decorators
> which impose different meanings, I'm stuck. Imagine I have two decorators

I imagine one nice way to approach the problem would be a
meta-decorator like the following:
[completely untested; please excuse likely Gmail line-wrapping]

NULL = object()
def annotation_sensitively_decorate(*decs):
    def decorate(f):
        annots = f.func_annotations
        keys = list(annots.keys())
        for dec, assignments in zip(decs, zip(*list(annots.values()))):
            # determine and swap in annotations for current decorator
            cur_annots = {k:v for k,v in zip(keys, assignments) if v
is not NULL}
            f.func_annotations = cur_annots
            f = dec(f)
        f.func_annotations = annots # restore orig annotations
        return f
    return decorate

@annotation_sensitively_decorate(log_me, memoize)
def c(foo: [do_log, memo_value(int)], bar: [no_log,
memo_value(tuple)], baz: [NULL, NULL]) -> [NULL, NULL]:
    ...

Cheers,
Chris
--
Metaprogramming is the best kind of programming, except when debugging.
http://blog.rebertia.com


From steve at pearwood.info  Sun Apr 17 10:10:22 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 17 Apr 2011 18:10:22 +1000
Subject: [Python-ideas] parameter annotation semantics
In-Reply-To: <BANLkTikMe2tt-0yd77RQgqdk0BvL+MwpGw@mail.gmail.com>
References: <BANLkTikMe2tt-0yd77RQgqdk0BvL+MwpGw@mail.gmail.com>
Message-ID: <4DAAA06E.1040209@pearwood.info>

Bruce Leban wrote:

> On Sat, Apr 16, 2011 at 6:27 PM, Steven D'Aprano <steve at pearwood.info>wrote:
> 
>> Bruce Leban wrote:
>>
>>  One
>>> of the strange aspects (to me) of parameter annotations is that they have
>>> no
>>> semantics which opens them up to multiple conflicting uses.
>>>
>> That's not a bug, that's a feature.
>>
>> It's been stated many times by Guido that it's far too early to standardize
>> on a single meaning for annotations. (We may *never* standardize on a single
>> meaning.) Instead, it is up to the library or decorator to impose whatever
>> meaning makes sense for that particular library or decorator.
> 
> 
> I understand that. I can still think it a bit strange, can't I? To be more
> specific, you hit the crux of the problem with the statement "it is up to *
> THE* library or decorator to impose whatever meaning makes sense" [emphasis
> added] is that it assumes the singular. If I want to use two decorators
> which impose different meanings, I'm stuck. Imagine I have two decorators
> like this:


Well, yes, you're stuck... but if we standardise on a single meaning, 
then you can't have two decorators with different meanings, can you?

This is kind like stating that we should standardise on a single flavour 
of ice cream (vanilla), because otherwise if you want two clashing 
flavours (liquorish and lemon, say) on the same cone, they will clash.

Yes, if you pick two clashing decorators, they will clash. So either 
write a bridge between them, or do without one.

This is hardly unique to annotations. You can't take two arbitrary 
decorators and sensibly combine them either.


@eat_cake_now
@save_cake_for_later
def get_cake():
     pass


Could developers write decorators that process annotations in a 
cooperative fashion, regardless of the nature of the other decorators? 
Perhaps they could, but that will hardly prevent clashes:

@exclude_given_types
@include_given_types
def func(arg: float):
     pass


But even ignoring such obvious clashes, I'm not convinced that such 
cooperative signature processing is desirable. It seems to me that to 
make this work, decorators would have to be written to ignore 
annotations they don't expect, Just In Case some other decorator might 
be expecting them:


@include_given_types
def func(arg: "float"):  # Oops, I meant a type object, not a string.
     pass


This would mask errors and make debugging much harder.


Now, of course this is not to suggest that any particular library can't 
be written in a cooperative fashion. But it seems to me that it would 
have to cooperate with a finite set of other *known* decorators (perhaps 
only its own, perhaps some other well-known library), rather than trying 
to interoperate with arbitrary decorators that expect arbitrary 
signatures and do arbitrary things.





-- 
Steven


From ncoghlan at gmail.com  Sun Apr 17 15:09:15 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Apr 2011 23:09:15 +1000
Subject: [Python-ideas] [Python-Dev] python and super
In-Reply-To: <4DA9D064.3050904@voidspace.org.uk>
References: <BANLkTimr4yWCCfgdm2KcWbpP-XcYP2SANw@mail.gmail.com>
	<70EF2C52-1D92-4351-884E-52AF76BAAC6D@mac.com>
	<4DA70ACA.4070204@voidspace.org.uk>
	<20110414153503.F125B3A4063@sparrow.telecommunity.com>
	<825E6CD5-8673-463B-92AE-59677C327C0A@gmail.com>
	<4DA71C63.3030809@voidspace.org.uk>
	<8A4A58EF-70F7-4F2F-8564-AE8611713986@mac.com>
	<BANLkTincGnYrc48jzQfNrc_HFidohJXvwA@mail.gmail.com>
	<4DA79E28.2060406@pearwood.info> <4DA84DD6.20608@voidspace.org.uk>
	<4DA8D6FC.9060707@canterbury.ac.nz>
	<4DA9D064.3050904@voidspace.org.uk>
Message-ID: <BANLkTikXFqSauCD1EQ22H9r=LC664DDXYA@mail.gmail.com>

On Sun, Apr 17, 2011 at 3:22 AM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
>
> No, not as I described. Where a method does not call up to its parent class
> then super would *not* auto call the parent class (I'm *not* suggesting a
> fully auto-call super as the original poster did), but a method not calling
> super still wouldn't halt the chain of calls. To achieve this a call into a
> method that doesn't call super (and would normally end the chain) instead
> removes itself and its unique parents [1] from the mro for this call.
>
> It would make the semantics more complex, but it would solve this problem
> and the original posters problem. For this specific example, if no classes
> call up to object.__init__ then it won't be called. It would permit you to
> have methods that can participate in multiple inheritance whilst still
> blocking calls (overriding) up to their parent classes.

Could you elaborate on the MRO data structure and super()
implementation changes you would propose in order to actually make it
possible to implement this idea?

The current MRO is calculated as a linear list at class definition
time (using the C3 algorithm), throwing away a lot of the original
tree information regarding the actual structure of the classes.

For example, the MRO "C, B, A, object" could come from a class
hierarchy that looked like:

class A: ...
class B(A): ...
class C(B): ...

Or one that looked like:

class A: ...
class B(A): ...
class C(B, A): ...

Or one that looked like:
class A: ...
class B: ...
class C(B, A): ...

If you add a "class D(C, B)" to the bottom of that hierarchy, then any
one of those 3 structures could apply across the parent classes, but D
would have the same MRO (i.e. "D, C, B, A, object"). And, of course,
there are now even *more* scenarios that could produce the same MRO
for D.

So if the only information you have available is D's MRO (which is the
current situation for super()), then you *don't know* whether B is C's
parent, sibling or both - the calculation of the MRO discards too much
detail about the class hierarchy.

Creating a dynamic MRO for every single method call just so people can
fail to think correctly about cooperative super() designs is a huge
price to pay to gain something that probably won't help in practice.

Auto-chaining of super calls just opens up a whole can of worms, and
if it can even be done at all, I definitely don't see how it could be
done in a backwards compatible way.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Sun Apr 17 15:13:58 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Apr 2011 23:13:58 +1000
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <BANLkTimhHupAzHWSCxQt+cYpvLpOoPECkw@mail.gmail.com>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
	<BANLkTimhHupAzHWSCxQt+cYpvLpOoPECkw@mail.gmail.com>
Message-ID: <BANLkTi=8X0Bf7cLy_Nra_tYzB=kHTZDWkg@mail.gmail.com>

On Sun, Apr 17, 2011 at 10:46 AM, Bruce Leban <bruce at leapyear.org> wrote:
> Note that I wrote?fd:[auto_init.private]?instead of?auto_init.private. One
> of the strange aspects?(to me)?of parameter annotations is that they have no
> semantics which opens them up to multiple conflicting uses. If we
> standardize on a convention that the annotation is a list (or tuple)
> of?annotations, then this leads us to usage like
>
> foo:[auto_init.name('bar'),constraint.non_negative,etc].

The idea is for annotations to be paired with decorators that define
the semantics.

Yes, that does mean that they aren't composable - decorators need to
provide an alternate initialisation mechanism for cases where the
annotations are already being used for something else.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From fuzzyman at voidspace.org.uk  Sun Apr 17 17:25:36 2011
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sun, 17 Apr 2011 16:25:36 +0100
Subject: [Python-ideas] [Python-Dev] python and super
In-Reply-To: <BANLkTikXFqSauCD1EQ22H9r=LC664DDXYA@mail.gmail.com>
References: <BANLkTimr4yWCCfgdm2KcWbpP-XcYP2SANw@mail.gmail.com>	<70EF2C52-1D92-4351-884E-52AF76BAAC6D@mac.com>	<4DA70ACA.4070204@voidspace.org.uk>	<20110414153503.F125B3A4063@sparrow.telecommunity.com>	<825E6CD5-8673-463B-92AE-59677C327C0A@gmail.com>	<4DA71C63.3030809@voidspace.org.uk>	<8A4A58EF-70F7-4F2F-8564-AE8611713986@mac.com>	<BANLkTincGnYrc48jzQfNrc_HFidohJXvwA@mail.gmail.com>	<4DA79E28.2060406@pearwood.info>	<4DA84DD6.20608@voidspace.org.uk>	<4DA8D6FC.9060707@canterbury.ac.nz>	<4DA9D064.3050904@voidspace.org.uk>
	<BANLkTikXFqSauCD1EQ22H9r=LC664DDXYA@mail.gmail.com>
Message-ID: <4DAB0670.5090504@voidspace.org.uk>

On 17/04/2011 14:09, Nick Coghlan wrote:
> On Sun, Apr 17, 2011 at 3:22 AM, Michael Foord
> <fuzzyman at voidspace.org.uk>  wrote:
>> No, not as I described. Where a method does not call up to its parent class
>> then super would *not* auto call the parent class (I'm *not* suggesting a
>> fully auto-call super as the original poster did), but a method not calling
>> super still wouldn't halt the chain of calls. To achieve this a call into a
>> method that doesn't call super (and would normally end the chain) instead
>> removes itself and its unique parents [1] from the mro for this call.
>>
>> It would make the semantics more complex, but it would solve this problem
>> and the original posters problem. For this specific example, if no classes
>> call up to object.__init__ then it won't be called. It would permit you to
>> have methods that can participate in multiple inheritance whilst still
>> blocking calls (overriding) up to their parent classes.
> Could you elaborate on the MRO data structure and super()
> implementation changes you would propose in order to actually make it
> possible to implement this idea?
>
> The current MRO is calculated as a linear list at class definition
> time (using the C3 algorithm), throwing away a lot of the original
> tree information regarding the actual structure of the classes.
>

Well, it isn't going to happen (no-one else is remotely in favour), so 
I'm not going to spend a lot more time on the topic. :-)

> For example, the MRO "C, B, A, object" could come from a class
> hierarchy that looked like:
>
> class A: ...
> class B(A): ...
> class C(B): ...
>
> Or one that looked like:
>
> class A: ...
> class B(A): ...
> class C(B, A): ...
>
> Or one that looked like:
> class A: ...
> class B: ...
> class C(B, A): ...
>
> If you add a "class D(C, B)" to the bottom of that hierarchy, then any
> one of those 3 structures could apply across the parent classes, but D
> would have the same MRO (i.e. "D, C, B, A, object"). And, of course,
> there are now even *more* scenarios that could produce the same MRO
> for D.
>
> So if the only information you have available is D's MRO (which is the
> current situation for super()), then you *don't know* whether B is C's
> parent, sibling or both - the calculation of the MRO discards too much
> detail about the class hierarchy.
>

But given that the mro contains the actual classes, *all* the 
information is available at runtime. (If a call to a method of B 
terminates the chain then super would be able to know what the base 
classes of B are by looking at B - even if that isn't expressed directly 
in the mro.)

> Creating a dynamic MRO for every single method call

It is *only* needed where you are mixing super calls with non super 
calls in the presence of multiple inheritance. So not for every method call.

> just so people can
> fail to think correctly about cooperative super() designs

I disagree with this categorisation. It is particularly for where you 
*are* thinking about cooperative super calls. The current algorithm has 
no way to express "don't call my parent class but continue the 
cooperative calls for other base classes".

This means that there are some multiple inheritance scenarios (like 
__init__ where more than one base class inherits directly from object) 
that you just can't do and use super. The alternative is not to use 
super at all, and for diamond inheritance that makes it hard to avoid 
calling up twice to some methods.

These are all *particularly* problems where you are mixing in classes 
that are from a third party library that you don't control. Even if 
these classes would be *perfectly amenable* for use as mixins via 
multiple inheritance, the current super semantics make that impossible. 
Using composition to get round this can be very messy and require custom 
code to do your dispatch, (again in diamond inheritance situations 
trying to avoid methods being called twice) when inheritance *could* 
just solve the problem (the problem from the original poster being one 
example of this).

Both of these are real problems, and a lot of the task of *explaining* 
super to Python programmers goes to explaining these caveats and when 
you can't use super. Yes I'm suggesting adding complexity to the 
implementation, but it should remove some of the complexity (and 
especially some of the painful caveats) when actually using it.

>   is a huge
> price to pay to gain something that probably won't help in practice.
>
> Auto-chaining of super calls just opens up a whole can of worms, and
> if it can even be done at all, I definitely don't see how it could be
> done in a backwards compatible way.

I'm not suggesting full auto-chaining. And especially given that the 
change only has any effect where you are mixing super calls with 
non-super calls, which you really can't do at the moment, I don't see 
any genuine backwards compatibility issues. If by backwards 
compatibility issues you mean preserving the current structure of 
__mro__, well it *could* be done without changing that - or by creating 
an alternative structure for multiple inheritance. Doesn't look like it 
will happen though. :-)

I'll probably blog about the idea for posterity, and then move on.

All the best,

Michael Foord

> Cheers,
> Nick.
>


-- 
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html



From greg.ewing at canterbury.ac.nz  Mon Apr 18 00:43:25 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 18 Apr 2011 10:43:25 +1200
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <4DAA4204.7020407@pearwood.info>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
	<BANLkTimhHupAzHWSCxQt+cYpvLpOoPECkw@mail.gmail.com>
	<4DAA4204.7020407@pearwood.info>
Message-ID: <4DAB6D0D.5040202@canterbury.ac.nz>

Steven D'Aprano wrote:

> That's not a bug, that's a feature.
> 
> It's been stated many times by Guido that it's far too early to 
> standardize on a single meaning for annotations.

I think it's a rather *strange* feature.

Imagine what would happen if someone other than Guido
proposed a syntax extension with no defined semantics
and no use cases. I don't think it would be very well
received!

-- 
Greg




From greg.ewing at canterbury.ac.nz  Mon Apr 18 00:57:28 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 18 Apr 2011 10:57:28 +1200
Subject: [Python-ideas] parameter annotation semantics
In-Reply-To: <BANLkTikMe2tt-0yd77RQgqdk0BvL+MwpGw@mail.gmail.com>
References: <BANLkTikMe2tt-0yd77RQgqdk0BvL+MwpGw@mail.gmail.com>
Message-ID: <4DAB7058.3050005@canterbury.ac.nz>

Bruce Leban wrote:

>     @memoize
>     @log_me
>     def c(foo:{memoize:memo_value(int)}, bar:{log_me:no_log,
>     memoize:memo_value(tuple)}):
>       pass
> 
> but I think that's too verbose and therefore less likely to be adopted.

Seems to me that *any* scheme for attaching multiple annotations
is likely to lead to unreadably verbose function headers. Or even
single annotations, for that matter.

Back when the decorator syntax was being hammered out, one of the
objections to putting the decorator on the same line as the function
header was that it could lead to excessively long and hard-to-read
headers. I think the whole open-slather annotation idea is asking
for the same thing in spades.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Mon Apr 18 01:35:31 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 18 Apr 2011 11:35:31 +1200
Subject: [Python-ideas] [Python-Dev] python and super
In-Reply-To: <4DAB0670.5090504@voidspace.org.uk>
References: <BANLkTimr4yWCCfgdm2KcWbpP-XcYP2SANw@mail.gmail.com>
	<70EF2C52-1D92-4351-884E-52AF76BAAC6D@mac.com>
	<4DA70ACA.4070204@voidspace.org.uk>
	<20110414153503.F125B3A4063@sparrow.telecommunity.com>
	<825E6CD5-8673-463B-92AE-59677C327C0A@gmail.com>
	<4DA71C63.3030809@voidspace.org.uk>
	<8A4A58EF-70F7-4F2F-8564-AE8611713986@mac.com>
	<BANLkTincGnYrc48jzQfNrc_HFidohJXvwA@mail.gmail.com>
	<4DA79E28.2060406@pearwood.info> <4DA84DD6.20608@voidspace.org.uk>
	<4DA8D6FC.9060707@canterbury.ac.nz> <4DA9D064.3050904@voidspace.org.uk>
	<BANLkTikXFqSauCD1EQ22H9r=LC664DDXYA@mail.gmail.com>
	<4DAB0670.5090504@voidspace.org.uk>
Message-ID: <4DAB7943.9020603@canterbury.ac.nz>

Michael Foord wrote:
> On 17/04/2011 14:09, Nick Coghlan wrote:

>> just so people can
>> fail to think correctly about cooperative super() designs

> I disagree with this categorisation. It is particularly for where you 
> *are* thinking about cooperative super calls.

I think what Nick means is that, although *you* might be
thinking about super calls, the people who wrote the classes
you're using did *not* (otherwise they would have included
the required super calls). So you're trying to mix classes
that weren't designed to be mixed together, which is likely
to lead to many more problems than just missing super calls.

> The current algorithm has 
> no way to express "don't call my parent class but continue the 
> cooperative calls for other base classes".

That's not quite what you're proposing. What you're proposing
is more like "call my parent class, but don't stop if my parent
class doesn't call *its* parent class."

Which still leaves the question of what happens if your parent's
parent turns up later in your own MRO. Your parent says not to
call it, but your MRO says to call it. Who wins, and why?

There's also the question of what to do with return values.
Without an answer to that, this feature would be restricted to
methods that don't return any useful value.

-- 
Greg


From cmjohnson.mailinglist at gmail.com  Mon Apr 18 01:54:26 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Sun, 17 Apr 2011 13:54:26 -1000
Subject: [Python-ideas] parameter annotation semantics
In-Reply-To: <4DAB7058.3050005@canterbury.ac.nz>
References: <BANLkTikMe2tt-0yd77RQgqdk0BvL+MwpGw@mail.gmail.com>
	<4DAB7058.3050005@canterbury.ac.nz>
Message-ID: <BANLkTim71U3bap1gXxLBMa7Laj1BNtm7Ow@mail.gmail.com>

On Sun, Apr 17, 2011 at 12:57 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Bruce Leban wrote:
>
>> ? ?@memoize
>> ? ?@log_me
>> ? ?def c(foo:{memoize:memo_value(int)}, bar:{log_me:no_log,
>> ? ?memoize:memo_value(tuple)}):
>> ? ? ?pass
>>
>> but I think that's too verbose and therefore less likely to be adopted.
>
> Seems to me that *any* scheme for attaching multiple annotations
> is likely to lead to unreadably verbose function headers. Or even
> single annotations, for that matter.
>
> Back when the decorator syntax was being hammered out, one of the
> objections to putting the decorator on the same line as the function
> header was that it could lead to excessively long and hard-to-read
> headers. I think the whole open-slather annotation idea is asking
> for the same thing in spades.

That's a good point, but the above example *could* be rewritten:

foo_ans = {memoize: memo_value(int)}
bar_ans = {log_me: no_log, memoize:memo_value(tuple)}

@memoize
@log_me
def c(foo: foo_ans, bar:bar_ans):
     ? ?pass

But at that point you're probably better off just making a fancy
decorator that takes parameters instead of stuffing everything into an
annotation.


From debatem1 at gmail.com  Mon Apr 18 02:45:01 2011
From: debatem1 at gmail.com (geremy condra)
Date: Sun, 17 Apr 2011 17:45:01 -0700
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <4DAB6D0D.5040202@canterbury.ac.nz>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>
	<BANLkTimhHupAzHWSCxQt+cYpvLpOoPECkw@mail.gmail.com>
	<4DAA4204.7020407@pearwood.info>
	<4DAB6D0D.5040202@canterbury.ac.nz>
Message-ID: <BANLkTi=2f9+J0HCQrxN4xpMoBJtg5euXag@mail.gmail.com>

On Sun, Apr 17, 2011 at 3:43 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Steven D'Aprano wrote:
>
>> That's not a bug, that's a feature.
>>
>> It's been stated many times by Guido that it's far too early to
>> standardize on a single meaning for annotations.
>
> I think it's a rather *strange* feature.
>
> Imagine what would happen if someone other than Guido
> proposed a syntax extension with no defined semantics
> and no use cases. I don't think it would be very well
> received!

I have a use case- I use them to wrap C functions using ctypes in a
semi-sane way.

Old way:

RAND_bytes = libraries['ssl'].RAND_bytes
RAND_bytes.restype = ctypes.c_int
RAND_bytes.argtypes = [ctypes.c_char_p, ctypes.c_int]

My way:

@C_function("ssl")
def RAND_bytes(iv: c_char_p, iv_length: c_int) -> c_int:
        return RAND_bytes.c_function(iv, iv_length)

IIRC, this came up during the initial discussion and wasn't widely
loved, but for me (I do a reasonable amount of glue work) it makes
life a lot simpler.

Geremy Condra


From steve at pearwood.info  Mon Apr 18 02:50:08 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 18 Apr 2011 10:50:08 +1000
Subject: [Python-ideas] Simple class initialization
In-Reply-To: <4DAB6D0D.5040202@canterbury.ac.nz>
References: <BANLkTimd1mLrCwHms5pqL=n7-v6RURh0ng@mail.gmail.com>	<BANLkTimhHupAzHWSCxQt+cYpvLpOoPECkw@mail.gmail.com>	<4DAA4204.7020407@pearwood.info>
	<4DAB6D0D.5040202@canterbury.ac.nz>
Message-ID: <4DAB8AC0.5080309@pearwood.info>

Greg Ewing wrote:
> Steven D'Aprano wrote:
> 
>> That's not a bug, that's a feature.
>>
>> It's been stated many times by Guido that it's far too early to 
>> standardize on a single meaning for annotations.
> 
> I think it's a rather *strange* feature.
> 
> Imagine what would happen if someone other than Guido
> proposed a syntax extension with no defined semantics
> and no use cases. I don't think it would be very well
> received!

The benefits of being BDFL :)

But seriously, check the PEP: it's not written by Guido, and this 
feature is not driven by whim.

http://www.python.org/dev/peps/pep-3107/

Annotations have at least one good use-case, and the semantics are 
perfectly defined: annotations are arbitrary expressions, and they get 
stored in the function object in a known place. What you do with those 
annotations is up to you. That's no different from other general 
processes in Python, like name binding, class attributes, and function 
calling, which have open semantics.

("Okay, I've created a variable. What do I do with it now?" That's 
entirely up to you, Python won't tell you what to do next.)

The only difference is that those other processes are so well-known and 
have existed in some cases since the dawn of time (Fortran), and so we 
take them for granted. Annotations in the Python sense are new, and 
nobody knows what to do with them yet (except for the oh-so-predictable 
idea of type testing -- boring!).

I think its a brave and innovative move. If it's not successful, that 
says more about the conservativeness of Python programmers than the 
usefulness of the feature. I bet Perl coders would have found some way 
to make their code even more incomprehensible with it by now *grin*

More here on type checking in Python here:

http://lambda-the-ultimate.org/node/1519

In particular note the links to Guido's essays thinking aloud, which 
eventually lead to annotations.



-- 
Steven



From ncoghlan at gmail.com  Mon Apr 18 04:30:26 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 18 Apr 2011 12:30:26 +1000
Subject: [Python-ideas] [Python-Dev] python and super
In-Reply-To: <4DAB7943.9020603@canterbury.ac.nz>
References: <BANLkTimr4yWCCfgdm2KcWbpP-XcYP2SANw@mail.gmail.com>
	<70EF2C52-1D92-4351-884E-52AF76BAAC6D@mac.com>
	<4DA70ACA.4070204@voidspace.org.uk>
	<20110414153503.F125B3A4063@sparrow.telecommunity.com>
	<825E6CD5-8673-463B-92AE-59677C327C0A@gmail.com>
	<4DA71C63.3030809@voidspace.org.uk>
	<8A4A58EF-70F7-4F2F-8564-AE8611713986@mac.com>
	<BANLkTincGnYrc48jzQfNrc_HFidohJXvwA@mail.gmail.com>
	<4DA79E28.2060406@pearwood.info> <4DA84DD6.20608@voidspace.org.uk>
	<4DA8D6FC.9060707@canterbury.ac.nz>
	<4DA9D064.3050904@voidspace.org.uk>
	<BANLkTikXFqSauCD1EQ22H9r=LC664DDXYA@mail.gmail.com>
	<4DAB0670.5090504@voidspace.org.uk>
	<4DAB7943.9020603@canterbury.ac.nz>
Message-ID: <BANLkTikLgwYggFcOQ_QftuMvc35am6hrAg@mail.gmail.com>

On Mon, Apr 18, 2011 at 9:35 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Michael Foord wrote:
>>
>> On 17/04/2011 14:09, Nick Coghlan wrote:
>
>>> just so people can
>>> fail to think correctly about cooperative super() designs
>
>> I disagree with this categorisation. It is particularly for where you
>> *are* thinking about cooperative super calls.
>
> I think what Nick means is that, although *you* might be
> thinking about super calls, the people who wrote the classes
> you're using did *not* (otherwise they would have included
> the required super calls). So you're trying to mix classes
> that weren't designed to be mixed together, which is likely
> to lead to many more problems than just missing super calls.

Yep, that's what I meant.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Mon Apr 18 04:47:11 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 18 Apr 2011 12:47:11 +1000
Subject: [Python-ideas] [Python-Dev] python and super
In-Reply-To: <4DAB0670.5090504@voidspace.org.uk>
References: <BANLkTimr4yWCCfgdm2KcWbpP-XcYP2SANw@mail.gmail.com>
	<70EF2C52-1D92-4351-884E-52AF76BAAC6D@mac.com>
	<4DA70ACA.4070204@voidspace.org.uk>
	<20110414153503.F125B3A4063@sparrow.telecommunity.com>
	<825E6CD5-8673-463B-92AE-59677C327C0A@gmail.com>
	<4DA71C63.3030809@voidspace.org.uk>
	<8A4A58EF-70F7-4F2F-8564-AE8611713986@mac.com>
	<BANLkTincGnYrc48jzQfNrc_HFidohJXvwA@mail.gmail.com>
	<4DA79E28.2060406@pearwood.info> <4DA84DD6.20608@voidspace.org.uk>
	<4DA8D6FC.9060707@canterbury.ac.nz>
	<4DA9D064.3050904@voidspace.org.uk>
	<BANLkTikXFqSauCD1EQ22H9r=LC664DDXYA@mail.gmail.com>
	<4DAB0670.5090504@voidspace.org.uk>
Message-ID: <BANLkTin_mY7XomxXQPocnCZRVhFb8Xn+=g@mail.gmail.com>

On Mon, Apr 18, 2011 at 1:25 AM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> On 17/04/2011 14:09, Nick Coghlan wrote:
>> So if the only information you have available is D's MRO (which is the
>> current situation for super()), then you *don't know* whether B is C's
>> parent, sibling or both - the calculation of the MRO discards too much
>> detail about the class hierarchy.
>>
>
> But given that the mro contains the actual classes, *all* the information is
> available at runtime. (If a call to a method of B terminates the chain then
> super would be able to know what the base classes of B are by looking at B -
> even if that isn't expressed directly in the mro.)

But super() is out of the picture by the time B returns - it's just an
object, it has no control over what happens when its methods return.
There's no overarching object with a view of the whole method chain,
just a series of object constructions and method calls:

obj.__mro__ == D, C, B, A, object

obj.method()
--> D.method(obj)
----> super(D, obj).method() # (aka C.method(obj))
------> super(C, obj).method() # (aka B.method(obj))
--------> super(B, obj).method() # (aka A.method(obj))
----------> super(A, obj).method() # (aka object.method(obj))

How does the caller of super(D, obj).method() know whether or not
super(C, obj).method() was called any more than the caller of
C.method(obj) knows whether or not B.method(obj) was called?

>> Creating a dynamic MRO for every single method call
>
> It is *only* needed where you are mixing super calls with non super calls in
> the presence of multiple inheritance. So not for every method call.

Then why build it into super() at all? Prove the concept by going back
to old-school super style and pass in the class reference explicitly
and create a dynamic MRO based on removal of your parent classes but
not your siblings. I just don't think it is possible to solve this as
simply as you appear to believe. If you can produce a working
prototype that actually has acceptable performance, is backwards
compatible with current cooperative super() code then I'll be happily
proven wrong, but the entire concept currently seems to be based on an
idea of iterating over the MRO in a way that *doesn't actually
happen*.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From grosser.meister.morti at gmx.net  Mon Apr 18 06:00:18 2011
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Mon, 18 Apr 2011 06:00:18 +0200
Subject: [Python-ideas] documentation: mention the version in which a syntax
 construct was added
Message-ID: <4DABB752.6020000@gmx.net>

I wonder in which version the "except Foo as e" syntax was added (as opposed to "except Foo, e"). 
The documentations does not state this but I think it wasn't there in Python 2.4.

I propose that such things are always mentioned in the documentation and formated so that you can 
easily find the version numbers.

See how Mozilla marks such things (scroll down):
https://developer.mozilla.org/en/DOM/range

	-panzi


From ben+python at benfinney.id.au  Mon Apr 18 06:08:10 2011
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 18 Apr 2011 14:08:10 +1000
Subject: [Python-ideas] documentation: mention the version in which a
	syntax construct was added
References: <4DABB752.6020000@gmx.net>
Message-ID: <87d3kkfbh1.fsf@benfinney.id.au>

Mathias Panzenb?ck <grosser.meister.morti at gmx.net>
writes:

> I wonder in which version the "except Foo as e" syntax was added (as
> opposed to "except Foo, e"). The documentations does not state this
> but I think it wasn't there in Python 2.4.

First in Python 3.0. Backported from there to Python 2.6.

<URL:http://www.python.org/dev/peps/pep-3110/#compatibility>

> I propose that such things are always mentioned in the documentation
> and formated so that you can easily find the version numbers.

Yes, that's a good proposal which is already known, and needs people to
do the work of implementing and maintaining it.

In the meantime, you have available the ?What's New In Python? documents
that accompany each version.

-- 
 \       ?The surest way to corrupt a youth is to instruct him to hold |
  `\       in higher esteem those who think alike than those who think |
_o__)             differently.? ?Friedrich Nietzsche, _The Dawn_, 1881 |
Ben Finney



From ncoghlan at gmail.com  Mon Apr 18 07:35:18 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 18 Apr 2011 15:35:18 +1000
Subject: [Python-ideas] documentation: mention the version in which a
 syntax construct was added
In-Reply-To: <4DABB752.6020000@gmx.net>
References: <4DABB752.6020000@gmx.net>
Message-ID: <BANLkTimwjwh6V+vJsGYGp46_puE2hsiKTg@mail.gmail.com>

On Mon, Apr 18, 2011 at 2:00 PM, Mathias Panzenb?ck
<grosser.meister.morti at gmx.net> wrote:
> I wonder in which version the "except Foo as e" syntax was added (as opposed
> to "except Foo, e"). The documentations does not state this but I think it
> wasn't there in Python 2.4.

It was new in 2.6 to match the way 3.x does it (see PEP 3110).

> I propose that such things are always mentioned in the documentation and
> formated so that you can easily find the version numbers.

That's actually a very good point. We're reasonably consistent about
doing this in the library reference (via the "versionadded" and
"versionchanged" tags), but it hasn't been done for the language
reference.

Fixing this would basically involve taking PEP 291, as well as the
What's New documents for 2.6 and 2.7, and using that information to
update the 2.7 language reference with appropriate annotations.

The comparable change for 3.x would use the 3.1 and 3.2 What's New to
update the 3.2 documentation (with a forward port to the 3.3
development line).

Keeping it in mind for future changes that affect the language
reference (such as PEP 380) is easy enough, but, as Ben noted, it will
require someone to submit the requisite patches in order to
retroactively add this information for past changes.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From tjreedy at udel.edu  Mon Apr 18 17:49:25 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 18 Apr 2011 11:49:25 -0400
Subject: [Python-ideas] documentation: mention the version in which a
 syntax construct was added
In-Reply-To: <BANLkTimwjwh6V+vJsGYGp46_puE2hsiKTg@mail.gmail.com>
References: <4DABB752.6020000@gmx.net>
	<BANLkTimwjwh6V+vJsGYGp46_puE2hsiKTg@mail.gmail.com>
Message-ID: <iohmi0$7l7$1@dough.gmane.org>

On 4/18/2011 1:35 AM, Nick Coghlan wrote:

> The comparable change for 3.x would use the 3.1 and 3.2 What's New to
> update the 3.2 documentation (with a forward port to the 3.3
> development line).

Because of the moratorium, there should not have been any syntax changes 
for 3.2, and I do not remember any for 3.1. I agree that version tags 
should be used in the language reference as well as the library reference.

-- 
Terry Jan Reedy



From masklinn at masklinn.net  Mon Apr 18 19:40:27 2011
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 18 Apr 2011 19:40:27 +0200
Subject: [Python-ideas] documentation: mention the version in which a
	syntax construct was added
In-Reply-To: <BANLkTimwjwh6V+vJsGYGp46_puE2hsiKTg@mail.gmail.com>
References: <4DABB752.6020000@gmx.net>
	<BANLkTimwjwh6V+vJsGYGp46_puE2hsiKTg@mail.gmail.com>
Message-ID: <B22840FB-86B0-48F4-B099-662248FFED44@masklinn.net>

On 2011-04-18, at 07:35 , Nick Coghlan wrote:
> On Mon, Apr 18, 2011 at 2:00 PM, Mathias Panzenb?ck
> <grosser.meister.morti at gmx.net> wrote:
>> I propose that such things are always mentioned in the documentation and
>> formated so that you can easily find the version numbers.
> 
> That's actually a very good point. We're reasonably consistent about
> doing this in the library reference (via the "versionadded" and
> "versionchanged" tags), but it hasn't been done for the language
> reference.
By the way, when we notice that the tag is missing in some places, what's the flow to fix it?

Today, I suggested using pkgutil.get_data to a colleague, and after a few checks I found out it was apparently added in 2.6 (it isn't available in my 2.5.5), but I didn't find that information marked up in the doc.

From tjreedy at udel.edu  Tue Apr 19 04:39:34 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 18 Apr 2011 22:39:34 -0400
Subject: [Python-ideas] documentation: mention the version in which a
 syntax construct was added
In-Reply-To: <B22840FB-86B0-48F4-B099-662248FFED44@masklinn.net>
References: <4DABB752.6020000@gmx.net>	<BANLkTimwjwh6V+vJsGYGp46_puE2hsiKTg@mail.gmail.com>
	<B22840FB-86B0-48F4-B099-662248FFED44@masklinn.net>
Message-ID: <ioisl6$q5r$1@dough.gmane.org>

On 4/18/2011 1:40 PM, Masklinn wrote:

> By the way, when we notice that the tag is missing in some places,
> what's the flow to fix it?

The tracker. Mark issue for 2.7, type documentation.
If you know of multiple omissions, put them altogether.

> Today, I suggested using pkgutil.get_data to a colleague, and after a
> few checks I found out it was apparently added in 2.6 (it isn't
> available in my 2.5.5), but I didn't find that information marked up
> in the doc.

Do mention (or quote from interpreter*) how you determine the info you 
request to be added. No many developers will want to go back and verify 
now just for 2.7 docs.

* something like
2.5.5
 >>> import pkgutil; pkgutil.get_data
...
Attribute error

2.6.x
 >>> import pkgutil; pkgutil.get_data
<Method type ob at ...>

-- 
Terry Jan Reedy



From blume.erich at gmail.com  Wed Apr 20 23:37:14 2011
From: blume.erich at gmail.com (Erich Blume)
Date: Wed, 20 Apr 2011 14:37:14 -0700
Subject: [Python-ideas] random.boolean or bernoulli
Message-ID: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>

I am yet a novice python user, and not a strong statistician to boot,
but I had an idea about how to enhance the 'random' module with one or
two new functions for generating boolean values instead of
floating-point values.

This idea has a lot of flexibility in how it might be implemented, but
I propose two new functions for the random module that might be
implemented in python as follows

def boolean():
    return choice((True,False))

def bernoulli(p):
    return random() <= p

It's true that since both of these functions have very simple
short-statement implementations that it might be unnecessary baggage,
but it occurred to me that their implementation might be consistent
with the rest of the module.

~eblume


From stefan_ml at behnel.de  Thu Apr 21 05:49:01 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 21 Apr 2011 05:49:01 +0200
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
Message-ID: <ioo9fe$nd6$1@dough.gmane.org>

Erich Blume, 20.04.2011 23:37:
> I am yet a novice python user, and not a strong statistician to boot,
> but I had an idea about how to enhance the 'random' module with one or
> two new functions for generating boolean values instead of
> floating-point values.
>
> This idea has a lot of flexibility in how it might be implemented, but
> I propose two new functions for the random module that might be
> implemented in python as follows
>
> def boolean():
>      return choice((True,False))

I like this one. It reads well in code:

     if random.boolean():
         ...

It reads less well with a from-import:

     if boolean():
         ...

But that's just a matter of renaming while importing it:

     from random import boolean as random_choice

     if random_choice():
         ...


> def bernoulli(p):
>      return random()<= p

This seems less obvious:

     if random.bernoulli(0.5):
         ...

Who's Random Bernoulli anyway?

Stefan



From raymond.hettinger at gmail.com  Thu Apr 21 05:55:05 2011
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 20 Apr 2011 20:55:05 -0700
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <ioo9fe$nd6$1@dough.gmane.org>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
	<ioo9fe$nd6$1@dough.gmane.org>
Message-ID: <871B0BA5-A8DB-47A8-8008-09B61E1C0E4B@gmail.com>


On Apr 20, 2011, at 8:49 PM, Stefan Behnel wrote:

> Erich Blume, 20.04.2011 23:37:
>> I am yet a novice python user, and not a strong statistician to boot,
>> but I had an idea about how to enhance the 'random' module with one or
>> two new functions for generating boolean values instead of
>> floating-point values.
>> 
>> This idea has a lot of flexibility in how it might be implemented, but
>> I propose two new functions for the random module that might be
>> implemented in python as follows
>> 
>> def boolean():
>>     return choice((True,False))
> 
> I like this one. It reads well in code:
> 
>    if random.boolean():
>        ...

The traditional way to spell it is:

   if random() < 0.5:
         ...


Raymond


From stefan_ml at behnel.de  Thu Apr 21 06:20:10 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 21 Apr 2011 06:20:10 +0200
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <871B0BA5-A8DB-47A8-8008-09B61E1C0E4B@gmail.com>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>	<ioo9fe$nd6$1@dough.gmane.org>
	<871B0BA5-A8DB-47A8-8008-09B61E1C0E4B@gmail.com>
Message-ID: <ioob9q$ul2$1@dough.gmane.org>

Raymond Hettinger, 21.04.2011 05:55:
> On Apr 20, 2011, at 8:49 PM, Stefan Behnel wrote:
>
>> Erich Blume, 20.04.2011 23:37:
>>> I am yet a novice python user, and not a strong statistician to boot,
>>> but I had an idea about how to enhance the 'random' module with one or
>>> two new functions for generating boolean values instead of
>>> floating-point values.
>>>
>>> This idea has a lot of flexibility in how it might be implemented, but
>>> I propose two new functions for the random module that might be
>>> implemented in python as follows
>>>
>>> def boolean():
>>>      return choice((True,False))
>>
>> I like this one. It reads well in code:
>>
>>     if random.boolean():
>>         ...
>
> The traditional way to spell it is:
>
>     if random()<  0.5:
>           ...

When I see constructs like this, my first thought is "Is there an 
off-by-one error here?", which then distracts my reading. It obviously 
wouldn't even matter here, since the randomness properties of random() are 
likely not good enough to see any difference, but that's second thought to 
me. It starts off by getting in my way.

Stefan



From raymond.hettinger at gmail.com  Thu Apr 21 07:11:49 2011
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 20 Apr 2011 22:11:49 -0700
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
Message-ID: <DB55F064-3CD9-4350-8C5F-D5805666FEDB@gmail.com>


On Apr 20, 2011, at 2:37 PM, Erich Blume wrote:

> I am yet a novice python user, and not a strong statistician to boot,
> but I had an idea about how to enhance the 'random' module with one or
> two new functions for generating boolean values instead of
> floating-point values.

ISTM, it would be better if you first gained some experience using the module as-is.


> This idea has a lot of flexibility in how it might be implemented, but
> I propose two new functions for the random module that might be
> implemented in python as follows
> 
> def boolean():
>    return choice((True,False))
> 
> def bernoulli(p):
>    return random() <= p
> 
> It's true that since both of these functions have very simple
> short-statement implementations that it might be unnecessary baggage,

I agree that they are unnecessary baggage.
AFAICT, other languages have avoided adding this sort of thing.
We already have randrange(), so this is just an inflexible specialization.
It is better to propose ideas that substantially increase the power of the module,
not ones that offer trivial respellings.


Raymond

From tjreedy at udel.edu  Thu Apr 21 09:38:49 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 21 Apr 2011 03:38:49 -0400
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <DB55F064-3CD9-4350-8C5F-D5805666FEDB@gmail.com>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
	<DB55F064-3CD9-4350-8C5F-D5805666FEDB@gmail.com>
Message-ID: <ioomu8$q41$1@dough.gmane.org>

On 4/21/2011 1:11 AM, Raymond Hettinger wrote:

>> It's true that since both of these functions have very simple
>> short-statement implementations that it might be unnecessary baggage,
>
> I agree that they are unnecessary baggage.
> AFAICT, other languages have avoided adding this sort of thing.
> We already have randrange(), so this is just an inflexible specialization.
> It is better to propose ideas that substantially increase the power of the module,
> not ones that offer trivial respellings.

Well put. -1 from me also.

-- 
Terry Jan Reedy



From greg.ewing at canterbury.ac.nz  Thu Apr 21 13:46:46 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 21 Apr 2011 23:46:46 +1200
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
Message-ID: <4DB01926.2070101@canterbury.ac.nz>

Erich Blume wrote:

> def bernoulli(p):
>     return random() <= p

When I use a function like this I've been calling
it chance(), which seems less jargony.

> def boolean():
>     return choice((True,False))

Since this is equal to chance(0.5), I'm not sure
it's worth it. A chance of exactly 50% seems like
a rather special case.

-- 
Greg


From jsbueno at python.org.br  Thu Apr 21 17:30:07 2011
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Thu, 21 Apr 2011 12:30:07 -0300
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <4DB01926.2070101@canterbury.ac.nz>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
	<4DB01926.2070101@canterbury.ac.nz>
Message-ID: <BANLkTin4Ko4+B0rpYh+8kCufnDsLO_3pAA@mail.gmail.com>

On Thu, Apr 21, 2011 at 8:46 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Erich Blume wrote:
>
>> def bernoulli(p):
>> ? ?return random() <= p
>
> When I use a function like this I've been calling
> it chance(), which seems less jargony.
>
>> def boolean():
>> ? ?return choice((True,False))
>
> Since this is equal to chance(0.5), I'm not sure
> it's worth it. A chance of exactly 50% seems like
> a rather special case.


What about  just:

def chance(n=0.5):
      return random() < n

?
As for "unnecessary baggage"  --about half the random module -
randrange, randint, uniform, choice,could be viewed as "unecessary
baggae" -- but the ability of cleamly specifying what one wants on the
random module through this functions, instead of having to deal with
the raw "random floating point from 0.0 to 1.0" as all other languages
is what probably made me start using Python, about 10 years ago.

  js
 -><-
>
> --
> Greg
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From stephen at xemacs.org  Thu Apr 21 19:02:00 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 22 Apr 2011 02:02:00 +0900
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <BANLkTin4Ko4+B0rpYh+8kCufnDsLO_3pAA@mail.gmail.com>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
	<4DB01926.2070101@canterbury.ac.nz>
	<BANLkTin4Ko4+B0rpYh+8kCufnDsLO_3pAA@mail.gmail.com>
Message-ID: <87k4enbks7.fsf@uwakimon.sk.tsukuba.ac.jp>

Joao S. O. Bueno writes:

 > What about  just:
 > 
 > def chance(n=0.5):
 >       return random() < n

n=0.5 just isn't that special.  Anyway, EIBTI.

Also, I almost certainly wouldn't bother to use random.chance() if it
existed.  I'd simply use application-specific definitions like

def random_gender():
    return 'female' if random() < p else 'male'

(it's quite rare that I actually want True and False as the values of
chance()-like functions).

 > As for "unnecessary baggage"  --about half the random module -
 > randrange, randint, uniform, choice,could be viewed as "unecessary
 > baggae"

Sure, this is always in the eye of the beholder.  My taste is that the
module gets it about right.  I've never needed the full functionality
of randrange, but I can imagine others using it fairly frequently, and
if I did need it it's cheaper to look it up than to code it up.  I do
often use "choice" and somewhat less often "shuffle".  These are
somewhat tedious to implement.

OTOH, to a mathematician, random() immediately makes one want to ask,
"what distribution?"  So uniform() really is needed.


From raymond.hettinger at gmail.com  Thu Apr 21 20:46:54 2011
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Thu, 21 Apr 2011 11:46:54 -0700
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <BANLkTin4Ko4+B0rpYh+8kCufnDsLO_3pAA@mail.gmail.com>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
	<4DB01926.2070101@canterbury.ac.nz>
	<BANLkTin4Ko4+B0rpYh+8kCufnDsLO_3pAA@mail.gmail.com>
Message-ID: <FF57EC2F-561B-40AC-89D9-3434B891AF11@gmail.com>


On Apr 21, 2011, at 8:30 AM, Joao S. O. Bueno wrote:
> 
> What about  just:
> 
> def chance(n=0.5):
>      return random() < n

Come on people.  This is junk and bad design.
Besides being unnecessary, trivial, opaque, and slow,
it has other issues like:
* no range checking for n<0 or n>1
* bad choice of argument name (p or x is used for probability
  while n is typically an integer representing a count)
* not obvious that it returns a boolean
* not obvious that a uniform distribution is presumed
* a name that will have difference interpretations for different
  people and make not make sense in a given context.
* no parallels in other languages (even Excel doesn't have this).
* it presumes that our users are not very bright
  and are in dire need of the language being dumbed down.

I'm amazed (and a little appalled) that the python-ideas crowd
would entertain adding this to a mature module like random.
Guido has had twenty years to put something like this in
the module (I believe he was the original writer) and likely
didn't do so for a good reason.  Even stats packages don't
seem to include anything this mundane.  The needs to be
some effort to not make modules unnecessarily fat and to
limit feature creep except for tools that greatly improve
expressive power.


Raymond


P.S.  Bernoulli isn't even jargon; it's a person's name.
A Bernoulli trial just means that events are independent.
It doesn't imply anything about a distribution or population
of possible result values.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110421/b826eafe/attachment.html>

From robert.kern at gmail.com  Thu Apr 21 21:11:15 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 21 Apr 2011 14:11:15 -0500
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <FF57EC2F-561B-40AC-89D9-3434B891AF11@gmail.com>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>	<4DB01926.2070101@canterbury.ac.nz>	<BANLkTin4Ko4+B0rpYh+8kCufnDsLO_3pAA@mail.gmail.com>
	<FF57EC2F-561B-40AC-89D9-3434B891AF11@gmail.com>
Message-ID: <iopvgk$ik0$1@dough.gmane.org>

On 4/21/11 1:46 PM, Raymond Hettinger wrote:

> P.S. Bernoulli isn't even jargon; it's a person's name.
> A Bernoulli trial just means that events are independent.
> It doesn't imply anything about a distribution or population
> of possible result values.

Actually, it is the canonical name of a particular discrete probability 
distribution. *If* one cared to add it, it would be a perfectly fine name for 
it, though "bernoullivariate" might fit better with the other named distributions.

   http://en.wikipedia.org/wiki/Bernoulli_distribution

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From mal at egenix.com  Thu Apr 21 22:25:09 2011
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 21 Apr 2011 22:25:09 +0200
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <iopvgk$ik0$1@dough.gmane.org>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>	<4DB01926.2070101@canterbury.ac.nz>	<BANLkTin4Ko4+B0rpYh+8kCufnDsLO_3pAA@mail.gmail.com>	<FF57EC2F-561B-40AC-89D9-3434B891AF11@gmail.com>
	<iopvgk$ik0$1@dough.gmane.org>
Message-ID: <4DB092A5.9020309@egenix.com>

Robert Kern wrote:
> On 4/21/11 1:46 PM, Raymond Hettinger wrote:
> 
>> P.S. Bernoulli isn't even jargon; it's a person's name.
>> A Bernoulli trial just means that events are independent.
>> It doesn't imply anything about a distribution or population
>> of possible result values.
> 
> Actually, it is the canonical name of a particular discrete probability
> distribution. *If* one cared to add it, it would be a perfectly fine
> name for it, though "bernoullivariate" might fit better with the other
> named distributions.
> 
>   http://en.wikipedia.org/wiki/Bernoulli_distribution
> 

I don't think those suggested functions are really missing from the
random module.

Distributions that are missing (at least AFAIK) are these two
important discrete distributions:

One which returns integers distributed according to the
binomial distribution B(n,p):

    http://en.wikipedia.org/wiki/Binomial_distribution

(The Bernoulli distribution is a special case (B(1,p)).)

The other important discrete distribution missing is the Poisson
distribution Pois(lambda):

    http://en.wikipedia.org/wiki/Poisson_distribution

Both are often used in simulations and test data generators
for real world discrete events.

In the past we've used Ivan Frohne's fine rv module for these,
but it would be great if they could be added to the standard
random module (taking benefit of the much better performance
this provides):

    http://svn.scipy.org/svn/scipy/tags/pre_numeric/Lib/stats/rv.py

Perhaps a nice project for a GSoC student...

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 21 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/


From anikom15 at gmail.com  Fri Apr 22 00:34:12 2011
From: anikom15 at gmail.com (Westley =?iso-8859-1?Q?Mart=EDnez?=)
Date: Thu, 21 Apr 2011 15:34:12 -0700
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <BANLkTin4Ko4+B0rpYh+8kCufnDsLO_3pAA@mail.gmail.com>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
	<4DB01926.2070101@canterbury.ac.nz>
	<BANLkTin4Ko4+B0rpYh+8kCufnDsLO_3pAA@mail.gmail.com>
Message-ID: <20110421223412.GA10971@Smoke>

On Thu, Apr 21, 2011 at 12:30:07PM -0300, Joao S. O. Bueno wrote:
> On Thu, Apr 21, 2011 at 8:46 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> [snip all this crap]

I find this idea to be more of a subroutine then a function.  They're
essentially the same but the principal is different.  If you would find
a random bool and random boyardee or whatever function useful in your
program, you're free to implement it for yourself.  That doesn't mean
we should add it to the stdlib, though.


From greg.ewing at canterbury.ac.nz  Fri Apr 22 02:17:34 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 22 Apr 2011 12:17:34 +1200
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <87k4enbks7.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
	<4DB01926.2070101@canterbury.ac.nz>
	<BANLkTin4Ko4+B0rpYh+8kCufnDsLO_3pAA@mail.gmail.com>
	<87k4enbks7.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <4DB0C91E.1070004@canterbury.ac.nz>

Stephen J. Turnbull wrote:
> I'd simply use application-specific definitions like
> 
> def random_gender():
>     return 'female' if random() < p else 'male'

The benefit of chance() is that it saves you from having
to think "Should that be < p or > p?", and readers of the
code thinking "Now does that mean a probability of p or
1-p?"

The answers to those questions might be second nature
to you, but it's not necessarily so for others.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Fri Apr 22 02:24:34 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 22 Apr 2011 12:24:34 +1200
Subject: [Python-ideas] random.boolean or bernoulli
In-Reply-To: <iopvgk$ik0$1@dough.gmane.org>
References: <BANLkTik1EMryCjVsRmUcCjP06J2hkag+0g@mail.gmail.com>
	<4DB01926.2070101@canterbury.ac.nz>
	<BANLkTin4Ko4+B0rpYh+8kCufnDsLO_3pAA@mail.gmail.com>
	<FF57EC2F-561B-40AC-89D9-3434B891AF11@gmail.com>
	<iopvgk$ik0$1@dough.gmane.org>
Message-ID: <4DB0CAC2.1040103@canterbury.ac.nz>

Robert Kern wrote:
> On 4/21/11 1:46 PM, Raymond Hettinger wrote:
> 
>> P.S. Bernoulli isn't even jargon;
> 
> Actually, it is the canonical name of a particular discrete probability 
> distribution. *If* one cared to add it, it would be a perfectly fine 
> name for it, though "bernoullivariate" might fit better with the other 
> named distributions.

The problem is that

  if bernoullivariate(0.5):
     do_something()

doesn't help to make the code any clearer to someone who
isn't steeped in statistical theory, whereas chance() has
a suggestive meaning to just about everyone.

-- 
Greg




From cool-rr at cool-rr.com  Fri Apr 22 11:59:39 2011
From: cool-rr at cool-rr.com (cool-RR)
Date: Fri, 22 Apr 2011 11:59:39 +0200
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
Message-ID: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>

Here's an idea that would have helped me today while coding. Allow something
like this:

    ('blue', 'red', 'orange' if some_condition, 'green')

So 'orange' is included in the tuple only if `some_condition` evaluates to
`True`. This could be applied to list literals, tuple literals, set
literals, and possibly dict literals, though the latter might be too clunky.

I expect this to be rejected, but I really couldn't think of an elegant way
to achieve the same thing with existing syntax.


Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110422/c4ad17c8/attachment.html>

From andreengels at gmail.com  Fri Apr 22 12:17:47 2011
From: andreengels at gmail.com (Andre Engels)
Date: Fri, 22 Apr 2011 12:17:47 +0200
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
Message-ID: <BANLkTimXK3pwcadD=2VWc+XN5H5Q-fY0oQ@mail.gmail.com>

On Fri, Apr 22, 2011 at 11:59 AM, cool-RR <cool-rr at cool-rr.com> wrote:
> Here's an idea that would have helped me today while coding. Allow something
> like this:
> ? ??('blue', 'red', 'orange' if some_condition, 'green')
> So 'orange' is included in the tuple only if `some_condition` evaluates to
> `True`. This could be applied to list literals, tuple literals, set
> literals, and possibly dict literals, though the latter might be too clunky.
> I expect this to be rejected, but I really couldn't think of an elegant way
> to achieve the same thing with existing syntax.

First note: If you can just remove an element in the middle and still
have the same time of object, then logically it would be a list, not
an array. But things that semantically are lists are indeed sometimes
made array for performance reasons, so that's not a weird thing to do.

Having said that, I don't think it's going to make it. Normally
there's exactly one element between each pair of commas; if we're
going to give up that, I'm not sure this is the best way to do it.

As for other ways to do it, they are not as elegant but they certainly do exist:

('blue', 'red') + (('orange',) if some_condition else ()) + 'green'

[c for c in ('blue', 'red', 'orange', 'green') if c != 'orange' or
some_condition]

-- 
Andr? Engels, andreengels at gmail.com


From python at mrabarnett.plus.com  Fri Apr 22 12:31:15 2011
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 22 Apr 2011 11:31:15 +0100
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTimXK3pwcadD=2VWc+XN5H5Q-fY0oQ@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<BANLkTimXK3pwcadD=2VWc+XN5H5Q-fY0oQ@mail.gmail.com>
Message-ID: <4DB158F3.90708@mrabarnett.plus.com>

On 22/04/2011 11:17, Andre Engels wrote:
> On Fri, Apr 22, 2011 at 11:59 AM, cool-RR<cool-rr at cool-rr.com>  wrote:
>> Here's an idea that would have helped me today while coding. Allow something
>> like this:
>>      ('blue', 'red', 'orange' if some_condition, 'green')
>> So 'orange' is included in the tuple only if `some_condition` evaluates to
>> `True`. This could be applied to list literals, tuple literals, set
>> literals, and possibly dict literals, though the latter might be too clunky.
>> I expect this to be rejected, but I really couldn't think of an elegant way
>> to achieve the same thing with existing syntax.
>
> First note: If you can just remove an element in the middle and still
> have the same time of object, then logically it would be a list, not
> an array. But things that semantically are lists are indeed sometimes
> made array for performance reasons, so that's not a weird thing to do.
>
> Having said that, I don't think it's going to make it. Normally
> there's exactly one element between each pair of commas; if we're
> going to give up that, I'm not sure this is the best way to do it.
>
> As for other ways to do it, they are not as elegant but they certainly do exist:
>
> ('blue', 'red') + (('orange',) if some_condition else ()) + 'green'
>
> [c for c in ('blue', 'red', 'orange', 'green') if c != 'orange' or
> some_condition]
>
You could write:

     def make_tuple(*items):
         return tuple(x for x in items if x is not None)

and then:

 >>> make_tuple('blue', 'red', 'orange' if some_condition else None, 
'green')
('blue', 'red', 'green')
 >>> some_condition = True
 >>> make_tuple('blue', 'red', 'orange' if some_condition else None, 
'green')
('blue', 'red', 'orange', 'green')


From ncoghlan at gmail.com  Fri Apr 22 12:42:39 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Apr 2011 20:42:39 +1000
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
Message-ID: <BANLkTi=NntdM-BAoA8om5K6VF8uinoeFAQ@mail.gmail.com>

On Fri, Apr 22, 2011 at 7:59 PM, cool-RR <cool-rr at cool-rr.com> wrote:
> Here's an idea that would have helped me today while coding. Allow something
> like this:
> ? ??('blue', 'red', 'orange' if some_condition, 'green')
> So 'orange' is included in the tuple only if `some_condition` evaluates to
> `True`. This could be applied to list literals, tuple literals, set
> literals, and possibly dict literals, though the latter might be too clunky.
> I expect this to be rejected, but I really couldn't think of an elegant way
> to achieve the same thing with existing syntax.

colours = 'blue red orange green'.split()
if not some_conditions:
    colours.remove('orange')

There are lots of options, but most of them start by not using a tuple
for a variable length sequence of like items.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From cool-rr at cool-rr.com  Fri Apr 22 12:48:33 2011
From: cool-rr at cool-rr.com (cool-RR)
Date: Fri, 22 Apr 2011 12:48:33 +0200
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTi=NntdM-BAoA8om5K6VF8uinoeFAQ@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<BANLkTi=NntdM-BAoA8om5K6VF8uinoeFAQ@mail.gmail.com>
Message-ID: <BANLkTikoXeWgN=-rRpk2Unr6oPnVeL-fhQ@mail.gmail.com>

On Fri, Apr 22, 2011 at 12:42 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Fri, Apr 22, 2011 at 7:59 PM, cool-RR <cool-rr at cool-rr.com> wrote:
> > Here's an idea that would have helped me today while coding. Allow
> something
> > like this:
> >     ('blue', 'red', 'orange' if some_condition, 'green')
> > So 'orange' is included in the tuple only if `some_condition` evaluates
> to
> > `True`. This could be applied to list literals, tuple literals, set
> > literals, and possibly dict literals, though the latter might be too
> clunky.
> > I expect this to be rejected, but I really couldn't think of an elegant
> way
> > to achieve the same thing with existing syntax.
>
> colours = 'blue red orange green'.split()
> if not some_conditions:
>    colours.remove('orange')
>
> There are lots of options, but most of them start by not using a tuple
> for a variable length sequence of like items.
>
> Cheers,
> Nick.


I see. It's about as elegant as the other suggestions. And it's pretty
annoying to use a list when I really wanted to use a tuple. Yeah, I can
convert it to a tuple at the end, but that's just making it more verbose.


Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110422/2d61abcd/attachment.html>

From andreengels at gmail.com  Fri Apr 22 12:52:05 2011
From: andreengels at gmail.com (Andre Engels)
Date: Fri, 22 Apr 2011 12:52:05 +0200
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTikoXeWgN=-rRpk2Unr6oPnVeL-fhQ@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<BANLkTi=NntdM-BAoA8om5K6VF8uinoeFAQ@mail.gmail.com>
	<BANLkTikoXeWgN=-rRpk2Unr6oPnVeL-fhQ@mail.gmail.com>
Message-ID: <BANLkTikvPxxcBEpxdF9y9OdvbDM3Es_VWQ@mail.gmail.com>

On Fri, Apr 22, 2011 at 12:48 PM, cool-RR <cool-rr at cool-rr.com> wrote:

> I see. It's about as elegant as the other suggestions. And it's pretty
> annoying to use a list when I really wanted to use a tuple. Yeah, I can
> convert it to a tuple at the end, but that's just making it more verbose.

But why do you want to have a tuple? A list is the logical Python
translation of this kind of semantics.


-- 
Andr? Engels, andreengels at gmail.com


From cmjohnson.mailinglist at gmail.com  Fri Apr 22 12:55:43 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Fri, 22 Apr 2011 00:55:43 -1000
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTikvPxxcBEpxdF9y9OdvbDM3Es_VWQ@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<BANLkTi=NntdM-BAoA8om5K6VF8uinoeFAQ@mail.gmail.com>
	<BANLkTikoXeWgN=-rRpk2Unr6oPnVeL-fhQ@mail.gmail.com>
	<BANLkTikvPxxcBEpxdF9y9OdvbDM3Es_VWQ@mail.gmail.com>
Message-ID: <BANLkTik5OBw0sy7NOE8H2gPWpByPm6Y1yA@mail.gmail.com>

On Fri, Apr 22, 2011 at 12:52 AM, Andre Engels <andreengels at gmail.com> wrote:
> On Fri, Apr 22, 2011 at 12:48 PM, cool-RR <cool-rr at cool-rr.com> wrote:
>
>> I see. It's about as elegant as the other suggestions. And it's pretty
>> annoying to use a list when I really wanted to use a tuple. Yeah, I can
>> convert it to a tuple at the end, but that's just making it more verbose.
>
> But why do you want to have a tuple? A list is the logical Python
> translation of this kind of semantics.

Exactly. Ignoring memory/performance differences, the semantic reason
to use a tuple instead of a list is that you have group of things _and
their ordering matters_. If you can arbitrarily omit some items in the
sequence, it's not a tuple; it's a list. (Or maybe even a set.)


From cool-rr at cool-rr.com  Fri Apr 22 12:57:12 2011
From: cool-rr at cool-rr.com (cool-RR)
Date: Fri, 22 Apr 2011 12:57:12 +0200
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTikvPxxcBEpxdF9y9OdvbDM3Es_VWQ@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<BANLkTi=NntdM-BAoA8om5K6VF8uinoeFAQ@mail.gmail.com>
	<BANLkTikoXeWgN=-rRpk2Unr6oPnVeL-fhQ@mail.gmail.com>
	<BANLkTikvPxxcBEpxdF9y9OdvbDM3Es_VWQ@mail.gmail.com>
Message-ID: <BANLkTinK-2yCaAO+L+MedFCYb4KSg_EN7A@mail.gmail.com>

On Fri, Apr 22, 2011 at 12:52 PM, Andre Engels <andreengels at gmail.com>wrote:

> On Fri, Apr 22, 2011 at 12:48 PM, cool-RR <cool-rr at cool-rr.com> wrote:
>
> > I see. It's about as elegant as the other suggestions. And it's pretty
> > annoying to use a list when I really wanted to use a tuple. Yeah, I can
> > convert it to a tuple at the end, but that's just making it more verbose.
>
> But why do you want to have a tuple? A list is the logical Python
> translation of this kind of semantics.
> --
> Andr? Engels, andreengels at gmail.com
>
>
"This kind of semantics"? What semantics did we have here except filtering
items before putting them in the sequence? How is that more list-y than
tuple-y? If I removed items *after* constructing the sequence, it'd be
list-y, but since I want to do it *before* the construction, I think it
doesn't make it list-y.

I used a tuple in this case because it's something that I wanted to stay
immutable.


Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110422/59cd1320/attachment.html>

From fdrake at acm.org  Fri Apr 22 13:39:54 2011
From: fdrake at acm.org (Fred Drake)
Date: Fri, 22 Apr 2011 07:39:54 -0400
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTinK-2yCaAO+L+MedFCYb4KSg_EN7A@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<BANLkTi=NntdM-BAoA8om5K6VF8uinoeFAQ@mail.gmail.com>
	<BANLkTikoXeWgN=-rRpk2Unr6oPnVeL-fhQ@mail.gmail.com>
	<BANLkTikvPxxcBEpxdF9y9OdvbDM3Es_VWQ@mail.gmail.com>
	<BANLkTinK-2yCaAO+L+MedFCYb4KSg_EN7A@mail.gmail.com>
Message-ID: <BANLkTinW0yp4A4xO-CFt17gkvfALo3534g@mail.gmail.com>

On Fri, Apr 22, 2011 at 6:57 AM, cool-RR <cool-rr at cool-rr.com> wrote:
> I used a tuple in this case because it's something that I wanted to stay
> immutable.

This is a perfect reason to use a tuple in my book.  A common and
handy approach with objects persisted in databases is to have the
persistent object track mutations, and make the attribute values
themselves immutable.  (I'm thinking ZODB in particular, but other
persistence implementations support similar models.)

Something I've often wanted to do is to construct a sequence (either
list or tuple) based on both constant and variable portions:

    >>> source = [...some list of things...]
    >>> result = [a, b, *(x for x in source if condition)]


  -Fred

-- 
Fred L. Drake, Jr.? ? <fdrake at acm.org>
"Give me the luxuries of life and I will willingly do without the necessities."
?? --Frank Lloyd Wright


From sturla at molden.no  Fri Apr 22 19:41:11 2011
From: sturla at molden.no (Sturla Molden)
Date: Fri, 22 Apr 2011 19:41:11 +0200
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
Message-ID: <4DB1BDB7.5020800@molden.no>

Den 22.04.2011 11:59, skrev cool-RR:
> Here's an idea that would have helped me today while coding. Allow 
> something like this:
>
>     ('blue', 'red', 'orange' if some_condition, 'green')
>
> So 'orange' is included in the tuple only if `some_condition` 
> evaluates to `True`.

This means it should be legal to write

     a = 'orange' if cond

which presumably should mean

      if cond: a = 'orange'

It retains some symmetry with

     a = 'orange' if cond else 'yellow'

Sturla












From bruce at leapyear.org  Fri Apr 22 20:44:04 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Fri, 22 Apr 2011 11:44:04 -0700
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <4DB1BDB7.5020800@molden.no>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<4DB1BDB7.5020800@molden.no>
Message-ID: <BANLkTinE0JuektUj3oUStBDWYo7hCFN3WA@mail.gmail.com>

On Fri, Apr 22, 2011 at 10:41 AM, Sturla Molden <sturla at molden.no> wrote:

> Den 22.04.2011 11:59, skrev cool-RR:
>
>  Here's an idea that would have helped me today while coding. Allow
>> something like this:
>>
>>    ('blue', 'red', 'orange' if some_condition, 'green')
>>
>> So 'orange' is included in the tuple only if `some_condition` evaluates to
>> `True`.
>>
>
> This means it should be legal to write
>
>    a = 'orange' if cond
>
> which presumably should mean
>
>     if cond: a = 'orange'
>

I don' think that having this as part of the tuple/list constructor means
that it "should be legal to write" everywhere else in the language. Would
you also expect:

f(a, b if c, d) == f(a, b, d) if c else f(a, d)

if foo if bar: == if bar: if foo:

return foo if bar == if bar: return foo


I fail to see an advantage in these while I do see that it would be useful
to write:

(a if x,
b if y,
c if z,
...)


If this works for tuple constructors, it *does* seem to me that it should
work for lists, sets and dicts. That last one is sticky is it:

{ a if x : 1 }
{ a : 1 if x }

or something else? Maybe tuples, lists and dicts are enough. Or maybe this
just isn't useful enough. I'm +0.1 on this.

--- Bruce
*New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April
Fools!
*New!** *Blog post: http://www.vroospeak.com Ironically, a glaring Google
grammatical error
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110422/e35c20e0/attachment.html>

From python at mrabarnett.plus.com  Fri Apr 22 21:00:00 2011
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 22 Apr 2011 20:00:00 +0100
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTinE0JuektUj3oUStBDWYo7hCFN3WA@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>	<4DB1BDB7.5020800@molden.no>
	<BANLkTinE0JuektUj3oUStBDWYo7hCFN3WA@mail.gmail.com>
Message-ID: <4DB1D030.2020201@mrabarnett.plus.com>

On 22/04/2011 19:44, Bruce Leban wrote:
>
>
> On Fri, Apr 22, 2011 at 10:41 AM, Sturla Molden <sturla at molden.no
> <mailto:sturla at molden.no>> wrote:
>
>     Den 22.04.2011 11:59, skrev cool-RR:
>
>         Here's an idea that would have helped me today while coding.
>         Allow something like this:
>
>             ('blue', 'red', 'orange' if some_condition, 'green')
>
>         So 'orange' is included in the tuple only if `some_condition`
>         evaluates to `True`.
>
>
>     This means it should be legal to write
>
>         a = 'orange' if cond
>
>     which presumably should mean
>
>          if cond: a = 'orange'
>
>
> I don' think that having this as part of the tuple/list constructor
> means that it "should be legal to write" everywhere else in the
> language. Would you also expect:
>
>     f(a, b if c, d) == f(a, b, d) if c else f(a, d)
>
Isn't the argument list of 'f' a tuple?

>     if foo if bar: == if bar: if foo:
>
>     return foo if bar == if bar: return foo
>
I'm having a bit of trouble parsing those...

>
> I fail to see an advantage in these while I do see that it would be
> useful to write:
>
>     (a if x,
>     b if y,
>     c if z,
>     ...)
>
>
> If this works for tuple constructors, it *does* seem to me that it
> should work for lists, sets and dicts. That last one is sticky is it:
>
>     { a if x : 1 }
>     { a : 1 if x }
>
The second one, because both the key and the value are conditional.

> or something else? Maybe tuples, lists and dicts are enough. Or maybe
> this just isn't useful enough. I'm +0.1 on this.
>


From guido at python.org  Fri Apr 22 21:06:06 2011
From: guido at python.org (Guido van Rossum)
Date: Fri, 22 Apr 2011 12:06:06 -0700
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <4DB1D030.2020201@mrabarnett.plus.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<4DB1BDB7.5020800@molden.no>
	<BANLkTinE0JuektUj3oUStBDWYo7hCFN3WA@mail.gmail.com>
	<4DB1D030.2020201@mrabarnett.plus.com>
Message-ID: <BANLkTik84TLVCFh=CQbAsyZBs9_7R18t2w@mail.gmail.com>

I'm -1 on this. It looks like a syntax error for

  'orange' if some_condition else 'another_color'

-- 
--Guido van Rossum (python.org/~guido)


From ericsnowcurrently at gmail.com  Fri Apr 22 21:07:05 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 22 Apr 2011 13:07:05 -0600
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <4DB1BDB7.5020800@molden.no>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<4DB1BDB7.5020800@molden.no>
Message-ID: <BANLkTi=w28k_Z8fc4w2gkO5eMri6ouBHQA@mail.gmail.com>

On Fri, Apr 22, 2011 at 11:41 AM, Sturla Molden <sturla at molden.no> wrote:

> Den 22.04.2011 11:59, skrev cool-RR:
>
>> Here's an idea that would have helped me today while coding. Allow
>> something like this:
>>
>>    ('blue', 'red', 'orange' if some_condition, 'green')
>>
>> So 'orange' is included in the tuple only if `some_condition` evaluates to
>> `True`.
>>
>
> This means it should be legal to write
>
>    a = 'orange' if cond
>
>
>
More accurately:

(a = 'orange') if cond

and such conditional assignment statements don't exist in python.


> which presumably should mean
>
>     if cond: a = 'orange'
>
> It retains some symmetry with
>
>    a = 'orange' if cond else 'yellow'
>
>
The symmetry is with the following:

a = 'orange' if cond else None

Though the else clause with None would be implicit.  This would result in
"a" getting set to one or the other.  However, the original idea implied
that the "a" should not be be set at all if cond is true, as results in your
example above with the if statement.

I am not sure about the feasibility of adding conditional execution clauses
to statements.  And then embedding a related conditional composition in
expression lists (for argument lists, etc.)...  I'm not sure it's worth it.


-eric


> Sturla
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110422/3bceebc5/attachment.html>

From bruce at leapyear.org  Fri Apr 22 21:08:36 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Fri, 22 Apr 2011 12:08:36 -0700
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <4DB1D030.2020201@mrabarnett.plus.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<4DB1BDB7.5020800@molden.no>
	<BANLkTinE0JuektUj3oUStBDWYo7hCFN3WA@mail.gmail.com>
	<4DB1D030.2020201@mrabarnett.plus.com>
Message-ID: <BANLkTikadaUkN+jfriZt1aFiDYzAJeopzg@mail.gmail.com>

On Fri, Apr 22, 2011 at 12:00 PM, MRAB <python at mrabarnett.plus.com> wrote:

>    if foo if bar: == if bar: if foo:
>>
>>    return foo if bar == if bar: return foo
>>
>>  I'm having a bit of trouble parsing those...


Sorry. I meant:

if foo if bar:

equivalent to:

if (foo if bar):

equivalent to:

if bar: if foo:

or more simply:

if bar and foo:


return foo if bar

equivalent to:

if bar: return foo


which I hope make it clear why I don't like them.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110422/c1d89a4a/attachment.html>

From sturla at molden.no  Fri Apr 22 22:47:02 2011
From: sturla at molden.no (Sturla Molden)
Date: Fri, 22 Apr 2011 22:47:02 +0200
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTi=w28k_Z8fc4w2gkO5eMri6ouBHQA@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>	<4DB1BDB7.5020800@molden.no>
	<BANLkTi=w28k_Z8fc4w2gkO5eMri6ouBHQA@mail.gmail.com>
Message-ID: <4DB1E946.1020003@molden.no>

Den 22.04.2011 21:07, skrev Eric Snow:
>
> a = 'orange' if cond else None
>

I like this interpretation:


a = 'orange' if cond else pass

(1, 2, 3, pass, 4) == (1,2,3,4)

pass,a,b = a,b



Sturla






From stephen at xemacs.org  Sat Apr 23 02:32:53 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 23 Apr 2011 09:32:53 +0900
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <4DB1E946.1020003@molden.no>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<4DB1BDB7.5020800@molden.no>
	<BANLkTi=w28k_Z8fc4w2gkO5eMri6ouBHQA@mail.gmail.com>
	<4DB1E946.1020003@molden.no>
Message-ID: <878vv1bydm.fsf@uwakimon.sk.tsukuba.ac.jp>

Sturla Molden writes:

 > I like this interpretation:

You mean "spelling", right?

 > a = 'orange' if cond else pass
 > 
 > (1, 2, 3, pass, 4) == (1,2,3,4)
 > 
 > pass,a,b = a,b

That's the only syntax proposed so far I like at all, but I still
can't imagine a common need for it that isn't served well enough by
the "tuple(comprehension)" spelling.  Maybe the OP (or somebody) can
expand a little on the use case for those of us lacking sufficient
imagination?


From anikom15 at gmail.com  Sat Apr 23 21:36:06 2011
From: anikom15 at gmail.com (Westley =?iso-8859-1?Q?Mart=EDnez?=)
Date: Sat, 23 Apr 2011 12:36:06 -0700
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <4DB1BDB7.5020800@molden.no>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<4DB1BDB7.5020800@molden.no>
Message-ID: <20110423193606.GB18157@Smoke>

On Fri, Apr 22, 2011 at 07:41:11PM +0200, Sturla Molden wrote:
> Den 22.04.2011 11:59, skrev cool-RR:
> >Here's an idea that would have helped me today while coding. Allow
> >something like this:
> >
> >    ('blue', 'red', 'orange' if some_condition, 'green')
> >
> >So 'orange' is included in the tuple only if `some_condition`
> >evaluates to `True`.
> 
> This means it should be legal to write
> 
>     a = 'orange' if cond
> 
> which presumably should mean
> 
>      if cond: a = 'orange'
> 
> It retains some symmetry with
> 
>     a = 'orange' if cond else 'yellow'
> 
> Sturla
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

No. This is a horrible idea.


From eric at trueblade.com  Sun Apr 24 12:15:33 2011
From: eric at trueblade.com (Eric Smith)
Date: Sun, 24 Apr 2011 06:15:33 -0400
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <20110409003950.6cb3eada@pitrou.net>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>	<20110408201807.6dc58ebf@pitrou.net>	<BANLkTikckxai4r5rQe0RCW30Z5+XrSEibg@mail.gmail.com>
	<20110409003950.6cb3eada@pitrou.net>
Message-ID: <4DB3F845.10608@trueblade.com>

> Guido van Rossum <guido at python.org> wrote:
>>On Fri, Apr 8, 2011 at 11:18 AM, Antoine Pitrou <solipsis at
pitrou.net> wrote:
>> On Fri, 8 Apr 2011 10:11:34 -0700
>> Guido van Rossum <guido at python.org> wrote:
>>> With apologies for not reading the PEP or this thread in full, some
comments:
>>>
>>> - I really like the syntax "except <exc> [as <var>] [if <test>]:".
>>> This addresses a pretty common use case in my experience. I don't care
>>> for the alternate proposed syntax that started this thread. I'm not
>>> sure that the 'if' subclause makes sense without the 'as' subclause,
>>> since most likely you'd want to refer to the caught exception. I note
>>> that it is more powerful than putting "if not <test>: raise" in the
>>> body of the except-clause, because it will allow subsequent except
>>> clauses to match still. I also note that it is a much "cleaner" change
>>> than (again) reorganizing the exception hierarchy, since there is no
>>> backward compatibility to consider.
>>
>> My main issue with said new syntax is that it doesn't make things much
>> easier to write.

>As I explained in other messages, it also adds semantics that are not
>so easily emulated with the existing syntax (you'd have to repeat code
>or use nested try/except blocks).

Interestingly, this is one of the few (only?) .NET features that is
exposed in Visual Basic but not in C#. Maybe there's something to learn
from that?

http://blogs.msdn.com/b/jaredpar/archive/2008/10/09/vb-catch-when-why-so-special.aspx

Eric.


From guido at python.org  Mon Apr 25 00:13:09 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 24 Apr 2011 15:13:09 -0700
Subject: [Python-ideas] PEP-3151 pattern-matching
In-Reply-To: <4DB3F845.10608@trueblade.com>
References: <BANLkTim722tCW5BSmf8VDAqQ8b-ye10TaQ@mail.gmail.com>
	<BANLkTikym_S5by0W1vR4dtWwaVLBmjOVZg@mail.gmail.com>
	<BANLkTinhjOj-MKnqJU2JSdD2UDSvyP=uGQ@mail.gmail.com>
	<BANLkTimv3H5WrHpwS8WeTbrtrxP4HJ4ndw@mail.gmail.com>
	<BANLkTi=+Zr884RRHSo32RuuUmaH2fxpuaA@mail.gmail.com>
	<20110408201807.6dc58ebf@pitrou.net>
	<BANLkTikckxai4r5rQe0RCW30Z5+XrSEibg@mail.gmail.com>
	<20110409003950.6cb3eada@pitrou.net> <4DB3F845.10608@trueblade.com>
Message-ID: <BANLkTi=d-THE8fwv_BqkzSWsa8+oBH2A6Q@mail.gmail.com>

On Sun, Apr 24, 2011 at 3:15 AM, Eric Smith <eric at trueblade.com> wrote:
>> Guido van Rossum <guido at python.org> wrote:
>>>On Fri, Apr 8, 2011 at 11:18 AM, Antoine Pitrou <solipsis at
> pitrou.net> wrote:
>>> On Fri, 8 Apr 2011 10:11:34 -0700
>>> Guido van Rossum <guido at python.org> wrote:
>>>> With apologies for not reading the PEP or this thread in full, some
> comments:
>>>>
>>>> - I really like the syntax "except <exc> [as <var>] [if <test>]:".
>>>> This addresses a pretty common use case in my experience. I don't care
>>>> for the alternate proposed syntax that started this thread. I'm not
>>>> sure that the 'if' subclause makes sense without the 'as' subclause,
>>>> since most likely you'd want to refer to the caught exception. I note
>>>> that it is more powerful than putting "if not <test>: raise" in the
>>>> body of the except-clause, because it will allow subsequent except
>>>> clauses to match still. I also note that it is a much "cleaner" change
>>>> than (again) reorganizing the exception hierarchy, since there is no
>>>> backward compatibility to consider.
>>>
>>> My main issue with said new syntax is that it doesn't make things much
>>> easier to write.
>
>>As I explained in other messages, it also adds semantics that are not
>>so easily emulated with the existing syntax (you'd have to repeat code
>>or use nested try/except blocks).
>
> Interestingly, this is one of the few (only?) .NET features that is
> exposed in Visual Basic but not in C#. Maybe there's something to learn
> from that?
>
> http://blogs.msdn.com/b/jaredpar/archive/2008/10/09/vb-catch-when-why-so-special.aspx

FWIW I still am at least +0.5 on the idea. I do note that currently
this is lecal syntax:

  try: ...
  except stuff if cond else otherstuff: ...

so allowing the syntax

  except <exception> [as <variable>] [if <condition>]:

could break some code; however

  except <exception> [as <variable> [if <condition>]]:

could not.

Anyone want to write a PEP (could be pretty short & sweet, the VB
reference would be useful) and a reference implementation?

-- 
--Guido van Rossum (python.org/~guido)


From sturla at molden.no  Mon Apr 25 01:18:17 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 25 Apr 2011 01:18:17 +0200
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <20110423193606.GB18157@Smoke>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>	<4DB1BDB7.5020800@molden.no>
	<20110423193606.GB18157@Smoke>
Message-ID: <4DB4AFB9.4030707@molden.no>

Den 23.04.2011 21:36, skrev Westley Mart?nez:
> No. This is a horrible idea.

Do you think so?  ;-)

The most "beautiful" thing with this syntax is this:

After typing

     foo = 'orange' if cond

or

     foo = 'orange' if cond else pass

we don't know if accessing foo will raise a NameError or
not (not without checking cond that is). And we all know
what an uncaught exception might do (cf. Ariane 5).


Sturla




From anikom15 at gmail.com  Mon Apr 25 04:03:59 2011
From: anikom15 at gmail.com (Westley =?iso-8859-1?Q?Mart=EDnez?=)
Date: Sun, 24 Apr 2011 19:03:59 -0700
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <4DB4AFB9.4030707@molden.no>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<4DB1BDB7.5020800@molden.no> <20110423193606.GB18157@Smoke>
	<4DB4AFB9.4030707@molden.no>
Message-ID: <20110425020359.GA27616@Smoke>

On Mon, Apr 25, 2011 at 01:18:17AM +0200, Sturla Molden wrote:
> Den 23.04.2011 21:36, skrev Westley Mart?nez:
> >No. This is a horrible idea.
> 
> Do you think so?  ;-)
> 
> The most "beautiful" thing with this syntax is this:
> 
> After typing
> 
>     foo = 'orange' if cond
> 
> or
> 
>     foo = 'orange' if cond else pass
> 
> we don't know if accessing foo will raise a NameError or
> not (not without checking cond that is). And we all know
> what an uncaught exception might do (cf. Ariane 5).
> 
> 
> Sturla
> 
> 

Right, the point of the else part of if/else statement is so we can
guarantee that foo is defined.


From nathan at cmu.edu  Mon Apr 25 05:34:23 2011
From: nathan at cmu.edu (Nathan Schneider)
Date: Sun, 24 Apr 2011 23:34:23 -0400
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <20110425020359.GA27616@Smoke>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<4DB1BDB7.5020800@molden.no> <20110423193606.GB18157@Smoke>
	<4DB4AFB9.4030707@molden.no> <20110425020359.GA27616@Smoke>
Message-ID: <BANLkTi=572s6ND4R+ADGgC6wT17pFNbNxw@mail.gmail.com>

My expectation would be that

  foo = 'orange' if cond

would be equivalent to

  foo = 'orange' if cond else None


Has this shorthand been considered before? It seems natural to me, and
I do find myself writing "else None" a lot, but from a (cursory)
search I couldn't find any discussion of it.

The only downside that comes to mind: conditionals in comprehensions
would look the same but have different semantics, i.e.

  [x for x in bar if f(x)]

might lead one to expect that None is added to the list wherever the
condition fails.


Nathan

On Sun, Apr 24, 2011 at 10:03 PM, Westley Mart?nez <anikom15 at gmail.com> wrote:
>
> On Mon, Apr 25, 2011 at 01:18:17AM +0200, Sturla Molden wrote:
> > Den 23.04.2011 21:36, skrev Westley Mart?nez:
> > >No. This is a horrible idea.
> >
> > Do you think so? ?;-)
> >
> > The most "beautiful" thing with this syntax is this:
> >
> > After typing
> >
> > ? ? foo = 'orange' if cond
> >
> > or
> >
> > ? ? foo = 'orange' if cond else pass
> >
> > we don't know if accessing foo will raise a NameError or
> > not (not without checking cond that is). And we all know
> > what an uncaught exception might do (cf. Ariane 5).
> >
> >
> > Sturla
> >
> >
>
> Right, the point of the else part of if/else statement is so we can
> guarantee that foo is defined.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas


From pyideas at rebertia.com  Mon Apr 25 06:20:55 2011
From: pyideas at rebertia.com (Chris Rebert)
Date: Sun, 24 Apr 2011 21:20:55 -0700
Subject: [Python-ideas] ('blue', 'red', 'orange' if something, 'green')
In-Reply-To: <BANLkTi=572s6ND4R+ADGgC6wT17pFNbNxw@mail.gmail.com>
References: <BANLkTin5kVkU0wq0gr1nw0=JEDHKaDB_aQ@mail.gmail.com>
	<4DB1BDB7.5020800@molden.no> <20110423193606.GB18157@Smoke>
	<4DB4AFB9.4030707@molden.no> <20110425020359.GA27616@Smoke>
	<BANLkTi=572s6ND4R+ADGgC6wT17pFNbNxw@mail.gmail.com>
Message-ID: <BANLkTim3tZ6CVuktVu1-mLTKZpsUgHkrSw@mail.gmail.com>

On Sun, Apr 24, 2011 at 8:34 PM, Nathan Schneider <nathan at cmu.edu> wrote:
> My expectation would be that
>
> ?foo = 'orange' if cond
>
> would be equivalent to
>
> ?foo = 'orange' if cond else None

Also equivalent:

foo = None
if cond: foo = 'orange'

> Has this shorthand been considered before? It seems natural to me, and
> I do find myself writing "else None" a lot, but from a (cursory)
> search I couldn't find any discussion of it.
>
> The only downside that comes to mind: conditionals in comprehensions
> would look the same but have different semantics, i.e.
>
> ?[x for x in bar if f(x)]
>
> might lead one to expect that None is added to the list wherever the
> condition fails.

Zen violations I perceive:

- Explicit is better than implicit.
    Is writing the extra 2 4-letter words, and thus being perfectly
clear, really that burdensome?

- Special cases aren't special enough to break the rules.
- There should be one-- and preferably only one --obvious way to do it.
    Down the route of adding trivial syntax sugar lies Perl.

- Errors should never pass silently.
    Something that is currently a syntax error (leaving off the `else
foo` part of a ternary expression) would now be a valid expression,
possibly leading to subtle typo-related bugs. Admittedly, how frequent
this error is in practice is questionable.

Please avoid top-posting in the future.

Cheers,
Chris
--
http://blog.rebertia.com


From haael at interia.pl  Mon Apr 25 21:36:39 2011
From: haael at interia.pl (haael)
Date: Mon, 25 Apr 2011 21:36:39 +0200
Subject: [Python-ideas] Make all keywords legal as an identifier
Message-ID: <4DB5CD47.3060808@interia.pl>


Hello, guys.

I did post this idea a few months ago. Now the revised version.


Goal:
Let _all_ alphanumeric keywords be legal as names for variables, functions and 
classes, even the ones that are reserved words now.

Rationale:
1. Python took most good English words as reserved tokens. Situation goes worse 
from version to version. I often have hard time searching for acceptable synonyms.
2. Because of that, old Python programs cease to work, even if they do not use 
any abandoned features. Their only sin is using certain words that further 
versions of Python have stolen away.
3. Sometimes one needs to import keywords from some other language, XML be an 
example, or "translate" another programming language into Python in one way or 
another. Keyword reservation is a big problem then; it does not allow to use 
the natural Python syntax.

Solution:
Let the parser treat all keywords that come after a dot (".") as regular 
identifiers.


For attributes, nothing changes:
 > boo.for = 7

For names that are not attributes, only one syntax change is needed: let a dot 
precede any identifier.
 > .with = 3

Of course, if a keyword is not preceded by a dot, it would be treated as a 
reserved word, just like now.
 > with = 3  # syntax error


There is only one case where a dot is used as a prefix of an identifier and 
that is a relative module import.
 > from .boo import foo
My change is consistent with this case.


One benefit would be that converting current programs to work with future 
versions would be a matter of simple grep.

Python is a great language. In my opinion, this change is the one last step to 
make it every geeky teenager's wet dream: the language where one can redefine 
almost anything. When I work with some problem, I always try to translate it to 
Python, solve and translate back. Prohibited identifier names are the main 
obstacle.

So, let's set the identifiers free and swallow all the world, making Python the 
least common denominator of every computer problem on this planet.


Regards,
Bartosz Tarnowski





-------------------------------------------------
Jedz te potrawy, aby uchronic sie przed rakiem!
Sprawdz  >> http://linkint.pl/f2946



From mikegraham at gmail.com  Mon Apr 25 21:43:42 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Mon, 25 Apr 2011 15:43:42 -0400
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB5CD47.3060808@interia.pl>
References: <4DB5CD47.3060808@interia.pl>
Message-ID: <BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>

On Mon, Apr 25, 2011 at 3:36 PM, haael <haael at interia.pl> wrote:
>
> Hello, guys.
>
> I did post this idea a few months ago. Now the revised version.
>
>
> Goal:
> Let _all_ alphanumeric keywords be legal as names for variables, functions
> and classes, even the ones that are reserved words now.
>
> Rationale:
> 1. Python took most good English words as reserved tokens. Situation goes
> worse from version to version. I often have hard time searching for
> acceptable synonyms.
This is exaggeration, but the point is well-taken.

> 2. Because of that, old Python programs cease to work, even if they do not
> use any abandoned features. Their only sin is using certain words that
> further versions of Python have stolen away.
You do not prevent this with your suggestion.

> 3. Sometimes one needs to import keywords from some other language, XML be
> an example, or "translate" another programming language into Python in one
> way or another. Keyword reservation is a big problem then; it does not allow
> to use the natural Python syntax.
>
> Solution:
> Let the parser treat all keywords that come after a dot (".") as regular
> identifiers.
>
>
> For attributes, nothing changes:
>> boo.for = 7
>
> For names that are not attributes, only one syntax change is needed: let a
> dot precede any identifier.
>> .with = 3
>
> Of course, if a keyword is not preceded by a dot, it would be treated as a
> reserved word, just like now.
>> with = 3 ?# syntax error
I don't see how this is a real improvement over the current
convention, to add a trailing underscore, so that programs really
needing to use the name "with" would use "with_". This does not
introduce any new syntax and maintains the same level of backwards
comparability your suggestion does.


> There is only one case where a dot is used as a prefix of an identifier and
> that is a relative module import.
>> from .boo import foo
> My change is consistent with this case.
>
>
> One benefit would be that converting current programs to work with future
> versions would be a matter of simple grep.
(This isn't actually true.)

> Python is a great language. In my opinion, this change is the one last step
> to make it every geeky teenager's wet dream: the language where one can
> redefine almost anything. When I work with some problem, I always try to
> translate it to Python, solve and translate back. Prohibited identifier
> names are the main obstacle.
>
> So, let's set the identifiers free and swallow all the world, making Python
> the least common denominator of every computer problem on this planet.
>
>
> Regards,
> Bartosz Tarnowski

Mike


From brian.curtin at gmail.com  Mon Apr 25 21:51:30 2011
From: brian.curtin at gmail.com (Brian Curtin)
Date: Mon, 25 Apr 2011 14:51:30 -0500
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB5CD47.3060808@interia.pl>
References: <4DB5CD47.3060808@interia.pl>
Message-ID: <BANLkTimKL6nyP+0Q33BMj209UU-3Z02-=Q@mail.gmail.com>

On Mon, Apr 25, 2011 at 14:36, haael <haael at interia.pl> wrote:

>
> Hello, guys.
>
> I did post this idea a few months ago. Now the revised version.
>
>
> Goal:
> Let _all_ alphanumeric keywords be legal as names for variables, functions
> and classes, even the ones that are reserved words now.
>
> Rationale:
> 1. Python took most good English words as reserved tokens. Situation goes
> worse from version to version. I often have hard time searching for
> acceptable synonyms.
> 2. Because of that, old Python programs cease to work, even if they do not
> use any abandoned features. Their only sin is using certain words that
> further versions of Python have stolen away.
> 3. Sometimes one needs to import keywords from some other language, XML be
> an example, or "translate" another programming language into Python in one
> way or another. Keyword reservation is a big problem then; it does not allow
> to use the natural Python syntax.
>
> Solution:
> Let the parser treat all keywords that come after a dot (".") as regular
> identifiers.
>
>
> For attributes, nothing changes:
> > boo.for = 7
>
> For names that are not attributes, only one syntax change is needed: let a
> dot precede any identifier.
> > .with = 3
>

Names tend to be nouns, so first I can't imagine why you'd want "with" as a
name, but you could exchange almost all keywords in the example and it's not
a great case. Making this change rather than working around poor name choice
gets a -1 from me.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110425/7a2fff1a/attachment.html>

From tjreedy at udel.edu  Mon Apr 25 22:02:28 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 25 Apr 2011 16:02:28 -0400
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB5CD47.3060808@interia.pl>
References: <4DB5CD47.3060808@interia.pl>
Message-ID: <ip4k0g$q05$1@dough.gmane.org>

On 4/25/2011 3:36 PM, haael wrote:

> 2. Because of that, old Python programs cease to work, even if they do
> not use any abandoned features. Their only sin is using certain words
> that further versions of Python have stolen away.

This very fact makes us *very* reluctant to add new keywords, which I 
think is a pretty good thing.

> Solution:
> Let the parser treat all keywords that come after a dot (".") as regular
> identifiers.

I have no idea what the technical difficulties would be. However, one 
possible practical difficulty is that whether or not a name is dotted 
depends on the context.

a.py:

x = 3 # ok
as = 4 # syntax error

b.py

import a
a.x = 3 # ok
a.as = 4 # ok with your proposal, but code within a could not access it.

Same for code in the body of a class statement versus code without.

-- 
Terry Jan Reedy



From mikegraham at gmail.com  Mon Apr 25 22:05:52 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Mon, 25 Apr 2011 16:05:52 -0400
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTimKL6nyP+0Q33BMj209UU-3Z02-=Q@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTimKL6nyP+0Q33BMj209UU-3Z02-=Q@mail.gmail.com>
Message-ID: <BANLkTike9TLrGW4oxes5-O3qCpcAwXaW9A@mail.gmail.com>

On Mon, Apr 25, 2011 at 3:51 PM, Brian Curtin <brian.curtin at gmail.com> wrote:
> On Mon, Apr 25, 2011 at 14:36, haael <haael at interia.pl> wrote:
>>
>> Hello, guys.
>>
>> I did post this idea a few months ago. Now the revised version.
>>
>>
>> Goal:
>> Let _all_ alphanumeric keywords be legal as names for variables, functions
>> and classes, even the ones that are reserved words now.
>>
>> Rationale:
>> 1. Python took most good English words as reserved tokens. Situation goes
>> worse from version to version. I often have hard time searching for
>> acceptable synonyms.
>> 2. Because of that, old Python programs cease to work, even if they do not
>> use any abandoned features. Their only sin is using certain words that
>> further versions of Python have stolen away.
>> 3. Sometimes one needs to import keywords from some other language, XML be
>> an example, or "translate" another programming language into Python in one
>> way or another. Keyword reservation is a big problem then; it does not allow
>> to use the natural Python syntax.
>>
>> Solution:
>> Let the parser treat all keywords that come after a dot (".") as regular
>> identifiers.
>>
>>
>> For attributes, nothing changes:
>> > boo.for = 7
>>
>> For names that are not attributes, only one syntax change is needed: let a
>> dot precede any identifier.
>> > .with = 3
>
> Names tend to be nouns, so first I can't imagine why you'd want "with" as a
> name, but you could exchange almost all keywords in the example and it's not
> a great case. Making this change rather than working around poor name choice
> gets a -1 from me.

To nitpick, names don't tend to be nouns only. Names of functions and
methods tend to be verbs and names of interfaces and abstract classes
are sometimes adjectives.

M


From sergio at gruposinternet.com.br  Mon Apr 25 21:53:19 2011
From: sergio at gruposinternet.com.br (=?ISO-8859-1?Q?S=E9rgio?= Surkamp)
Date: Mon, 25 Apr 2011 16:53:19 -0300
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB5CD47.3060808@interia.pl>
References: <4DB5CD47.3060808@interia.pl>
Message-ID: <20110425165319.345a160c@icedearth.corp.grupos.com.br>

> For names that are not attributes, only one syntax change is needed:
> let a dot precede any identifier.
>  > .with = 3

Why don't you use underscore instead of a dot?

_with = 3

Regards,
-- 
  .:''''':.
.:'        `     S?rgio Surkamp | Gerente de Rede
::    ........   sergio at gruposinternet.com.br
`:.        .:'
  `:,   ,.:'     *Grupos Internet S.A.*
    `: :'        R. Lauro Linhares, 2123 Torre B - Sala 201
     : :         Trindade - Florian?polis - SC
     :.'
     ::          +55 48 3234-4109
     :
     '           http://www.gruposinternet.com.br


From brian.curtin at gmail.com  Mon Apr 25 22:13:53 2011
From: brian.curtin at gmail.com (Brian Curtin)
Date: Mon, 25 Apr 2011 15:13:53 -0500
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTike9TLrGW4oxes5-O3qCpcAwXaW9A@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTimKL6nyP+0Q33BMj209UU-3Z02-=Q@mail.gmail.com>
	<BANLkTike9TLrGW4oxes5-O3qCpcAwXaW9A@mail.gmail.com>
Message-ID: <BANLkTininhh5B8WCubc_3TRCJLx7=zRjQA@mail.gmail.com>

On Mon, Apr 25, 2011 at 15:05, Mike Graham <mikegraham at gmail.com> wrote:

> On Mon, Apr 25, 2011 at 3:51 PM, Brian Curtin <brian.curtin at gmail.com>
> wrote:
> > On Mon, Apr 25, 2011 at 14:36, haael <haael at interia.pl> wrote:
> >>
> >> Hello, guys.
> >>
> >> I did post this idea a few months ago. Now the revised version.
> >>
> >>
> >> Goal:
> >> Let _all_ alphanumeric keywords be legal as names for variables,
> functions
> >> and classes, even the ones that are reserved words now.
> >>
> >> Rationale:
> >> 1. Python took most good English words as reserved tokens. Situation
> goes
> >> worse from version to version. I often have hard time searching for
> >> acceptable synonyms.
> >> 2. Because of that, old Python programs cease to work, even if they do
> not
> >> use any abandoned features. Their only sin is using certain words that
> >> further versions of Python have stolen away.
> >> 3. Sometimes one needs to import keywords from some other language, XML
> be
> >> an example, or "translate" another programming language into Python in
> one
> >> way or another. Keyword reservation is a big problem then; it does not
> allow
> >> to use the natural Python syntax.
> >>
> >> Solution:
> >> Let the parser treat all keywords that come after a dot (".") as regular
> >> identifiers.
> >>
> >>
> >> For attributes, nothing changes:
> >> > boo.for = 7
> >>
> >> For names that are not attributes, only one syntax change is needed: let
> a
> >> dot precede any identifier.
> >> > .with = 3
> >
> > Names tend to be nouns, so first I can't imagine why you'd want "with" as
> a
> > name, but you could exchange almost all keywords in the example and it's
> not
> > a great case. Making this change rather than working around poor name
> choice
> > gets a -1 from me.
>
> To nitpick, names don't tend to be nouns only. Names of functions and
> methods tend to be verbs and names of interfaces and abstract classes
> are sometimes adjectives.


The point still stands regardless of my your nitpicking. "with" is a bad
name for any of those. As is try, raise, pass, import, break, True, etc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110425/09a1086d/attachment.html>

From masklinn at masklinn.net  Mon Apr 25 22:21:07 2011
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 25 Apr 2011 22:21:07 +0200
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTininhh5B8WCubc_3TRCJLx7=zRjQA@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTimKL6nyP+0Q33BMj209UU-3Z02-=Q@mail.gmail.com>
	<BANLkTike9TLrGW4oxes5-O3qCpcAwXaW9A@mail.gmail.com>
	<BANLkTininhh5B8WCubc_3TRCJLx7=zRjQA@mail.gmail.com>
Message-ID: <BADC24AD-AF0C-405E-8B9D-CAA739B28828@masklinn.net>

On 2011-04-25, at 22:13 , Brian Curtin wrote:
> On Mon, Apr 25, 2011 at 15:05, Mike Graham <mikegraham at gmail.com> wrote:
>> On Mon, Apr 25, 2011 at 3:51 PM, Brian Curtin <brian.curtin at gmail.com>
>> wrote:
>>> On Mon, Apr 25, 2011 at 14:36, haael <haael at interia.pl> wrote:
>>>> 
>>>> Hello, guys.
>>>> 
>>>> I did post this idea a few months ago. Now the revised version.
>>>> 
>>>> 
>>>> Goal:
>>>> Let _all_ alphanumeric keywords be legal as names for variables,
>> functions
>>>> and classes, even the ones that are reserved words now.
>>>> 
>>>> Rationale:
>>>> 1. Python took most good English words as reserved tokens. Situation
>> goes
>>>> worse from version to version. I often have hard time searching for
>>>> acceptable synonyms.
>>>> 2. Because of that, old Python programs cease to work, even if they do
>> not
>>>> use any abandoned features. Their only sin is using certain words that
>>>> further versions of Python have stolen away.
>>>> 3. Sometimes one needs to import keywords from some other language, XML
>> be
>>>> an example, or "translate" another programming language into Python in
>> one
>>>> way or another. Keyword reservation is a big problem then; it does not
>> allow
>>>> to use the natural Python syntax.
>>>> 
>>>> Solution:
>>>> Let the parser treat all keywords that come after a dot (".") as regular
>>>> identifiers.
>>>> 
>>>> 
>>>> For attributes, nothing changes:
>>>>> boo.for = 7
>>>> 
>>>> For names that are not attributes, only one syntax change is needed: let
>> a
>>>> dot precede any identifier.
>>>>> .with = 3
>>> 
>>> Names tend to be nouns, so first I can't imagine why you'd want "with" as
>> a
>>> name, but you could exchange almost all keywords in the example and it's
>> not
>>> a great case. Making this change rather than working around poor name
>> choice
>>> gets a -1 from me.
>> 
>> To nitpick, names don't tend to be nouns only. Names of functions and
>> methods tend to be verbs and names of interfaces and abstract classes
>> are sometimes adjectives.
> 
> 
> The point still stands regardless of my your nitpicking. "with" is a bad
> name for any of those. As is try, raise, pass, import, break, True, etc.

With is actually a very nice name for some things, it creates very readable, english-looking code.

And what about `class`? Or `for` (that one clashes hard against the HTML object model, label elements have a for attribute). `in`, `except` or `is` may also be interesting in some cases.

Do all Python keywords have this issue? No, I doubt anybody's ever tried to called an attribute `elif`, but I definitely ran into the issue a few times.

From ethan at stoneleaf.us  Mon Apr 25 22:35:35 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Apr 2011 13:35:35 -0700
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTininhh5B8WCubc_3TRCJLx7=zRjQA@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>	<BANLkTimKL6nyP+0Q33BMj209UU-3Z02-=Q@mail.gmail.com>	<BANLkTike9TLrGW4oxes5-O3qCpcAwXaW9A@mail.gmail.com>
	<BANLkTininhh5B8WCubc_3TRCJLx7=zRjQA@mail.gmail.com>
Message-ID: <4DB5DB17.2010600@stoneleaf.us>

Brian Curtin wrote:
> On Mon, Apr 25, 2011 at 15:05, Mike Graham wrote:
>> On Mon, Apr 25, 2011 at 3:51 PM, Brian Curtin wrote:
>>> Names tend to be nouns, so first I can't imagine why you'd want "with" as a
>>> name, but you could exchange almost all keywords in the example and it's not
>>> a great case. Making this change rather than working around poor name choice
>>> gets a -1 from me.
> 
>> To nitpick, names don't tend to be nouns only. Names of functions and
>> methods tend to be verbs and names of interfaces and abstract classes
>> are sometimes adjectives.
> 
> The point still stands regardless of my your nitpicking. "with" is a bad 
> name for any of those. As is try, raise, pass, import, break, True, etc.
> 

I can easily see several of those working fine for boolean names.

Still -1 to the idea, though.

~Ethan~


From mikegraham at gmail.com  Mon Apr 25 22:26:47 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Mon, 25 Apr 2011 16:26:47 -0400
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTininhh5B8WCubc_3TRCJLx7=zRjQA@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTimKL6nyP+0Q33BMj209UU-3Z02-=Q@mail.gmail.com>
	<BANLkTike9TLrGW4oxes5-O3qCpcAwXaW9A@mail.gmail.com>
	<BANLkTininhh5B8WCubc_3TRCJLx7=zRjQA@mail.gmail.com>
Message-ID: <BANLkTinU3LAajpFk0aJUPD6tvwkgnPqH7w@mail.gmail.com>

On Mon, Apr 25, 2011 at 4:13 PM, Brian Curtin <brian.curtin at gmail.com> wrote:
> On Mon, Apr 25, 2011 at 15:05, Mike Graham <mikegraham at gmail.com> wrote:
>>
>> On Mon, Apr 25, 2011 at 3:51 PM, Brian Curtin <brian.curtin at gmail.com>
>> wrote:
>> > On Mon, Apr 25, 2011 at 14:36, haael <haael at interia.pl> wrote:
>> >>
>> >> Hello, guys.
>> >>
>> >> I did post this idea a few months ago. Now the revised version.
>> >>
>> >>
>> >> Goal:
>> >> Let _all_ alphanumeric keywords be legal as names for variables,
>> >> functions
>> >> and classes, even the ones that are reserved words now.
>> >>
>> >> Rationale:
>> >> 1. Python took most good English words as reserved tokens. Situation
>> >> goes
>> >> worse from version to version. I often have hard time searching for
>> >> acceptable synonyms.
>> >> 2. Because of that, old Python programs cease to work, even if they do
>> >> not
>> >> use any abandoned features. Their only sin is using certain words that
>> >> further versions of Python have stolen away.
>> >> 3. Sometimes one needs to import keywords from some other language, XML
>> >> be
>> >> an example, or "translate" another programming language into Python in
>> >> one
>> >> way or another. Keyword reservation is a big problem then; it does not
>> >> allow
>> >> to use the natural Python syntax.
>> >>
>> >> Solution:
>> >> Let the parser treat all keywords that come after a dot (".") as
>> >> regular
>> >> identifiers.
>> >>
>> >>
>> >> For attributes, nothing changes:
>> >> > boo.for = 7
>> >>
>> >> For names that are not attributes, only one syntax change is needed:
>> >> let a
>> >> dot precede any identifier.
>> >> > .with = 3
>> >
>> > Names tend to be nouns, so first I can't imagine why you'd want "with"
>> > as a
>> > name, but you could exchange almost all keywords in the example and it's
>> > not
>> > a great case. Making this change rather than working around poor name
>> > choice
>> > gets a -1 from me.
>>
>> To nitpick, names don't tend to be nouns only. Names of functions and
>> methods tend to be verbs and names of interfaces and abstract classes
>> are sometimes adjectives.
>
> The point still stands regardless of my your nitpicking. "with" is a bad
> name for any of those. As is try, raise, pass, import, break, True, etc.

I get what you're saying, but it's not categorically the case for
Python keywords.

For example:
 - To use your list, the throw method of generators should be raise.
 - The assertTrue method of TestCase should be assert.
 - The first argument of a classmethod should be class.
 - sqlalbemy.and_ and or_ should be and and or.
 - In an example snippet, "for a, b, c in zip(as, bs, cs)" would be
nice. Similarly for is.

Other examples, some more contrived than others, could be
provided--some of these would be good names if not for their keyword
status. However, I don't think I've seen a suggestion better than the
current solution (or lack thereof).

Mike


From mikegraham at gmail.com  Mon Apr 25 22:27:43 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Mon, 25 Apr 2011 16:27:43 -0400
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTinU3LAajpFk0aJUPD6tvwkgnPqH7w@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTimKL6nyP+0Q33BMj209UU-3Z02-=Q@mail.gmail.com>
	<BANLkTike9TLrGW4oxes5-O3qCpcAwXaW9A@mail.gmail.com>
	<BANLkTininhh5B8WCubc_3TRCJLx7=zRjQA@mail.gmail.com>
	<BANLkTinU3LAajpFk0aJUPD6tvwkgnPqH7w@mail.gmail.com>
Message-ID: <BANLkTimZ7+FJHvz7=Tpt4Yzk0TKfyEmjhw@mail.gmail.com>

On Mon, Apr 25, 2011 at 4:26 PM, Mike Graham <mikegraham at gmail.com> wrote:
> On Mon, Apr 25, 2011 at 4:13 PM, Brian Curtin <brian.curtin at gmail.com> wrote:
>> On Mon, Apr 25, 2011 at 15:05, Mike Graham <mikegraham at gmail.com> wrote:
>>>
>>> On Mon, Apr 25, 2011 at 3:51 PM, Brian Curtin <brian.curtin at gmail.com>
>>> wrote:
>>> > On Mon, Apr 25, 2011 at 14:36, haael <haael at interia.pl> wrote:
>>> >>
>>> >> Hello, guys.
>>> >>
>>> >> I did post this idea a few months ago. Now the revised version.
>>> >>
>>> >>
>>> >> Goal:
>>> >> Let _all_ alphanumeric keywords be legal as names for variables,
>>> >> functions
>>> >> and classes, even the ones that are reserved words now.
>>> >>
>>> >> Rationale:
>>> >> 1. Python took most good English words as reserved tokens. Situation
>>> >> goes
>>> >> worse from version to version. I often have hard time searching for
>>> >> acceptable synonyms.
>>> >> 2. Because of that, old Python programs cease to work, even if they do
>>> >> not
>>> >> use any abandoned features. Their only sin is using certain words that
>>> >> further versions of Python have stolen away.
>>> >> 3. Sometimes one needs to import keywords from some other language, XML
>>> >> be
>>> >> an example, or "translate" another programming language into Python in
>>> >> one
>>> >> way or another. Keyword reservation is a big problem then; it does not
>>> >> allow
>>> >> to use the natural Python syntax.
>>> >>
>>> >> Solution:
>>> >> Let the parser treat all keywords that come after a dot (".") as
>>> >> regular
>>> >> identifiers.
>>> >>
>>> >>
>>> >> For attributes, nothing changes:
>>> >> > boo.for = 7
>>> >>
>>> >> For names that are not attributes, only one syntax change is needed:
>>> >> let a
>>> >> dot precede any identifier.
>>> >> > .with = 3
>>> >
>>> > Names tend to be nouns, so first I can't imagine why you'd want "with"
>>> > as a
>>> > name, but you could exchange almost all keywords in the example and it's
>>> > not
>>> > a great case. Making this change rather than working around poor name
>>> > choice
>>> > gets a -1 from me.
>>>
>>> To nitpick, names don't tend to be nouns only. Names of functions and
>>> methods tend to be verbs and names of interfaces and abstract classes
>>> are sometimes adjectives.
>>
>> The point still stands regardless of my your nitpicking. "with" is a bad
>> name for any of those. As is try, raise, pass, import, break, True, etc.
>
> I get what you're saying, but it's not categorically the case for
> Python keywords.
>
> For example:
> ?- To use your list, the throw method of generators should be raise.
> ?- The assertTrue method of TestCase should be assert.
> ?- The first argument of a classmethod should be class.
> ?- sqlalbemy.and_ and or_ should be and and or.
> ?- In an example snippet, "for a, b, c in zip(as, bs, cs)" would be
> nice. Similarly for is.
>
> Other examples, some more contrived than others, could be
> provided--some of these would be good names if not for their keyword
> status. However, I don't think I've seen a suggestion better than the
> current solution (or lack thereof).
>
> Mike
>
By "current solution" I mean "what Python does now (nothing, you can't
use the names)". I do not think this proposal should be implemented.

MG


From haael at interia.pl  Mon Apr 25 22:33:54 2011
From: haael at interia.pl (haael)
Date: Mon, 25 Apr 2011 22:33:54 +0200
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>
Message-ID: <4DB5DAB2.6090305@interia.pl>


@ Mike Graham

>> Of course, if a keyword is not preceded by a dot, it would be treated as a
>> reserved word, just like now.
>>> with = 3  # syntax error
> I don't see how this is a real improvement over the current
> convention, to add a trailing underscore, so that programs really
> needing to use the name "with" would use "with_". This does not
> introduce any new syntax and maintains the same level of backwards
> comparability your suggestion does.

But the trailing underscore is treated as a part of an identifier, while the 
preceding dot is not. This is important if I want to have an identifier named 
exactly "with", with no other characters (no pun itended).

As I said, I want sometimes to import some non-Python namespace, i.e. a Pascal 
program. If all identifiers are allowed, there would never be a clash of 
reserved words.



@ Terry Reedy

 > This very fact makes us *very* reluctant to add new keywords, which I think
 > is a pretty good thing.

So my change hits two birds with one stone: programmers could use any word as 
an identifier, developers could use any word as a token. Perfect solution.

 > as = 4 # syntax error

Read my proposal carefully. The module could access this name with a preceding dot:

 > .as = 4 # access to global and local variables



@ Sergio Surkamp

 > Why don't you use underscore instead of a dot?

As I said, the underscore is a part of a name, while the dot isn't.



@ Brain Curtin

 > Names tend to be nouns, so first I can't imagine why you'd want "with" as a
 > name, but you could exchange almost all keywords in the example and it's not
 > a great case. Making this change rather than working around poor name choice
 > gets a -1 from me.

First of all, many nouns are reserved, i.e. "object" or "class".

Second: variable names are usually nouns indeed, but functions and methods are 
often verbs, while named parameters can be prepositions and adverbs.

For example:
 > turtles.fight(with=honour)

Python kidnapped many verbs and prepositions and made them reserved.

However, no matter what we say, it's the programmer's choice which word to use. 
If he has a reason to use prepositions as variable names, it's none of our 
business.



Regards,
Bartosz Tarnowski



---------------------------------------------
Ksiegowa radzi: Jak za??ozyc firme w 15 minut?
http://linkint.pl/f2968



From ethan at stoneleaf.us  Mon Apr 25 23:26:05 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Apr 2011 14:26:05 -0700
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB5DAB2.6090305@interia.pl>
References: <4DB5CD47.3060808@interia.pl>	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>
	<4DB5DAB2.6090305@interia.pl>
Message-ID: <4DB5E6ED.40904@stoneleaf.us>

haael wrote:
> @ Brain Curtin
> 
>> Names tend to be nouns, so first I can't imagine why you'd want "with" as a
>> name, but you could exchange almost all keywords in the example and it's not
>> a great case. Making this change rather than working around poor name choice
>> gets a -1 from me.
> 
> First of all, many nouns are reserved, i.e. "object" or "class".

Many?  Aren't we still at less than 50 words total?  Pretty 
infinitesimal when compared with the 100,000+ words in the English language.

> Second: variable names are usually nouns indeed, but functions and 
> methods are often verbs, while named parameters can be prepositions and 
> adverbs.
> 
> For example:
>> turtles.fight(with=honour)
> 
> Python kidnapped many verbs and prepositions and made them reserved.

See above.  This is a ridiculous exaggeration.

~Ethan~


From cmjohnson.mailinglist at gmail.com  Tue Apr 26 01:53:36 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Mon, 25 Apr 2011 13:53:36 -1000
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB5E6ED.40904@stoneleaf.us>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>
	<4DB5DAB2.6090305@interia.pl> <4DB5E6ED.40904@stoneleaf.us>
Message-ID: <BANLkTimLQ_sNdCqEo+Ek5vSJJhXGTssyLQ@mail.gmail.com>

On Mon, Apr 25, 2011 at 11:26 AM, Ethan Furman <ethan at stoneleaf.us> wrote:


> Many? ?Aren't we still at less than 50 words total? ?Pretty infinitesimal
> when compared with the 100,000+ words in the English language.

Here are all of them for Python 3:


and                 elif                import              raise
as                  else                in                  return
assert              except              is                  try
break               finally             lambda              while
class               for                 nonlocal            with
continue            from                not                 yield
def                 global              or
del                 if                  pass

30 total.

I say, append a _ and you're fine.


From pyideas at rebertia.com  Tue Apr 26 02:14:55 2011
From: pyideas at rebertia.com (Chris Rebert)
Date: Mon, 25 Apr 2011 17:14:55 -0700
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTimLQ_sNdCqEo+Ek5vSJJhXGTssyLQ@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>
	<4DB5DAB2.6090305@interia.pl> <4DB5E6ED.40904@stoneleaf.us>
	<BANLkTimLQ_sNdCqEo+Ek5vSJJhXGTssyLQ@mail.gmail.com>
Message-ID: <BANLkTin3ZTgGoQBSiGfW+28z91gc36KpJA@mail.gmail.com>

On Mon, Apr 25, 2011 at 4:53 PM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
> On Mon, Apr 25, 2011 at 11:26 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>> Many? ?Aren't we still at less than 50 words total? ?Pretty infinitesimal
>> when compared with the 100,000+ words in the English language.
>
> Here are all of them for Python 3:
>
>
> and ? ? ? ? ? ? ? ? elif ? ? ? ? ? ? ? ?import ? ? ? ? ? ? ?raise
> as ? ? ? ? ? ? ? ? ?else ? ? ? ? ? ? ? ?in ? ? ? ? ? ? ? ? ?return
> assert ? ? ? ? ? ? ?except ? ? ? ? ? ? ?is ? ? ? ? ? ? ? ? ?try
> break ? ? ? ? ? ? ? finally ? ? ? ? ? ? lambda ? ? ? ? ? ? ?while
> class ? ? ? ? ? ? ? for ? ? ? ? ? ? ? ? nonlocal ? ? ? ? ? ?with
> continue ? ? ? ? ? ?from ? ? ? ? ? ? ? ?not ? ? ? ? ? ? ? ? yield
> def ? ? ? ? ? ? ? ? global ? ? ? ? ? ? ?or
> del ? ? ? ? ? ? ? ? if ? ? ? ? ? ? ? ? ?pass
>
> 30 total.

You're missing True, False, and None.

Ellipsis remains an interesting exception.

Cheers,
Chris

From rob.cliffe at btinternet.com  Tue Apr 26 02:52:53 2011
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Tue, 26 Apr 2011 01:52:53 +0100
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTin3ZTgGoQBSiGfW+28z91gc36KpJA@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>	<4DB5DAB2.6090305@interia.pl>
	<4DB5E6ED.40904@stoneleaf.us>	<BANLkTimLQ_sNdCqEo+Ek5vSJJhXGTssyLQ@mail.gmail.com>
	<BANLkTin3ZTgGoQBSiGfW+28z91gc36KpJA@mail.gmail.com>
Message-ID: <4DB61765.4070809@btinternet.com>



On 26/04/2011 01:14, Chris Rebert wrote:
> On Mon, Apr 25, 2011 at 4:53 PM, Carl M. Johnson
> <cmjohnson.mailinglist at gmail.com>  wrote:
>> On Mon, Apr 25, 2011 at 11:26 AM, Ethan Furman<ethan at stoneleaf.us>  wrote:
>>> Many?  Aren't we still at less than 50 words total?  Pretty infinitesimal
>>> when compared with the 100,000+ words in the English language.
>> Here are all of them for Python 3:
>>
>>
>> and                 elif                import              raise
>> as                  else                in                  return
>> assert              except              is                  try
>> break               finally             lambda              while
>> class               for                 nonlocal            with
>> continue            from                not                 yield
>> def                 global              or
>> del                 if                  pass
>>
>> 30 total.
> You're missing True, False, and None.
>
> Ellipsis remains an interesting exception.
>
> Cheers,
> Chris
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
If you don't like adding an underscore, why not capitalise the first 
Letter of your Identifiers, e.g. 'With'?  This would be fine for 
importing from Pascal (where Identifiers are Case-insensitive) and would 
guarantee avoiding clashing with Keywords, present *and* future (even if 
it gave a slightly German Flavour to the Code).
Rob Cliffe



From wickedgrey at gmail.com  Tue Apr 26 03:35:47 2011
From: wickedgrey at gmail.com (Eli Stevens (Gmail))
Date: Mon, 25 Apr 2011 18:35:47 -0700
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB5CD47.3060808@interia.pl>
References: <4DB5CD47.3060808@interia.pl>
Message-ID: <BANLkTimqCoebEg2ZNhBA47OmUSpGo3yKmg@mail.gmail.com>

with .with.as as .as:
    .assert(.as.if(.not) for .not in .for.not.in if .import(.not.None,
.as, .None))

I don't think that this improves program readability or clarity.

Eli


From stephen at xemacs.org  Tue Apr 26 03:53:11 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 26 Apr 2011 10:53:11 +0900
Subject: [Python-ideas]  Make all keywords legal as an identifier
In-Reply-To: <4DB5CD47.3060808@interia.pl>
References: <4DB5CD47.3060808@interia.pl>
Message-ID: <87oc3taid4.fsf@uwakimon.sk.tsukuba.ac.jp>

haael writes:

 > Goal:
 > Let _all_ alphanumeric keywords be legal as names for variables,
 > functions and classes, even the ones that are reserved words now.

-1

Do you have any idea how many TB of hate mail you will get from
maintainers of Python modes for editors if this goes through?<wink>

Philosophically, Lewis Carroll put "paid" to this proposal more than
100 years ago.  All the king's horses and all the king's men won't be
able to put it together again.

To be specific, having reserved tokens in the syntax, including
keywords, enhances readability.  Python's #1 feature is readability.
Let's keep it that way!



From cmjohnson.mailinglist at gmail.com  Tue Apr 26 03:50:50 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Mon, 25 Apr 2011 15:50:50 -1000
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTin3ZTgGoQBSiGfW+28z91gc36KpJA@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>
	<4DB5DAB2.6090305@interia.pl> <4DB5E6ED.40904@stoneleaf.us>
	<BANLkTimLQ_sNdCqEo+Ek5vSJJhXGTssyLQ@mail.gmail.com>
	<BANLkTin3ZTgGoQBSiGfW+28z91gc36KpJA@mail.gmail.com>
Message-ID: <BANLkTim87KhxAyhb9V99VNmOuB9K9g9NSw@mail.gmail.com>

On Mon, Apr 25, 2011 at 2:14 PM, Chris Rebert <pyideas at rebertia.com> wrote:
> On Mon, Apr 25, 2011 at 4:53 PM, Carl M. Johnson
>> Here are all of them for Python 3:
>>
>>
>> and ? ? ? ? ? ? ? ? elif ? ? ? ? ? ? ? ?import ? ? ? ? ? ? ?raise
>> as ? ? ? ? ? ? ? ? ?else ? ? ? ? ? ? ? ?in ? ? ? ? ? ? ? ? ?return
>> assert ? ? ? ? ? ? ?except ? ? ? ? ? ? ?is ? ? ? ? ? ? ? ? ?try
>> break ? ? ? ? ? ? ? finally ? ? ? ? ? ? lambda ? ? ? ? ? ? ?while
>> class ? ? ? ? ? ? ? for ? ? ? ? ? ? ? ? nonlocal ? ? ? ? ? ?with
>> continue ? ? ? ? ? ?from ? ? ? ? ? ? ? ?not ? ? ? ? ? ? ? ? yield
>> def ? ? ? ? ? ? ? ? global ? ? ? ? ? ? ?or
>> del ? ? ? ? ? ? ? ? if ? ? ? ? ? ? ? ? ?pass
>>
>> 30 total.
>
> You're missing True, False, and None.

Looks like a bug in help("keywords") under Python 3.2.

From cmjohnson.mailinglist at gmail.com  Tue Apr 26 03:57:16 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Mon, 25 Apr 2011 15:57:16 -1000
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTim87KhxAyhb9V99VNmOuB9K9g9NSw@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>
	<4DB5DAB2.6090305@interia.pl> <4DB5E6ED.40904@stoneleaf.us>
	<BANLkTimLQ_sNdCqEo+Ek5vSJJhXGTssyLQ@mail.gmail.com>
	<BANLkTin3ZTgGoQBSiGfW+28z91gc36KpJA@mail.gmail.com>
	<BANLkTim87KhxAyhb9V99VNmOuB9K9g9NSw@mail.gmail.com>
Message-ID: <BANLkTikP7Of=cJ7p3ecUwgdMZphoKNQLpg@mail.gmail.com>

> Looks like a bug in help("keywords") under Python 3.2.

Filed it: http://bugs.python.org/issue11926


From steve at pearwood.info  Tue Apr 26 04:35:38 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 26 Apr 2011 12:35:38 +1000
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB5DAB2.6090305@interia.pl>
References: <4DB5CD47.3060808@interia.pl>	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>
	<4DB5DAB2.6090305@interia.pl>
Message-ID: <4DB62F7A.3050201@pearwood.info>

haael wrote:

> As I said, I want sometimes to import some non-Python namespace, i.e. a 
> Pascal program. If all identifiers are allowed, there would never be a 
> clash of reserved words.

How do you import non-Python code? I don't understand this argument.

[...]
> So my change hits two birds with one stone: programmers could use any 
> word as an identifier, developers could use any word as a token. Perfect 
> solution.

On the contrary -- this would actually be a bad thing, in my opinion 
significantly worse than the current situation. I accept that name 
clashes can be a problem, but I dispute that it is a big problem. It's a 
tiny, minuscule problem, and I argue that your solution is much worse.

You are only considering the *cost* of having reserved words and not the 
*benefit*. Your proposal increases the cognitive load on the programmer, 
rather than decreases it. We write code once, but we read it many times, 
and it is much more important to minimize the burden of reading code 
than writing it.

The rule now is simple: there are a small number of reserved words. You 
may not use them, end of story. Therefore you can identify a reserved 
word *immediately* you see it, without caring about content beyond "is 
it inside a string or a comment?". The cognitive burden on the reader is 
very low.

You would make that rule more complex. Reserved words would no longer be 
reserved. That alone is a bad thing -- you want to solve the problem 
that you can't name your variables "return", "for", "if", but I don't 
call that a problem, I call that a GOOD thing. I don't want to read code 
that sometimes uses return as a statement and sometimes as a variable.

But worse, things which were a clear mistake before become ambiguous:

len(.spam)

is clearly an unambiguous mistake now, but with your proposal it becomes 
valid syntax. Now I have to stop and think: did the author mean .spam or 
did he mean obj.spam and just forget the obj part?

If I don't use the dotted form, then Python might "steal" or "kidnap" 
(your words, not mine) my identifier, so if I care about forward 
compatibility, I will use dots everywhere:

class .tree(.parent):
     def .func(self, .argument, .exception=None):
         if not .exception:
              return self.method(.argument, .param)
         raise .exception

The cognitive load on reading my code is increased. The dots don't 
actually do anything, they're just there to protect my code from 
distant, *hypothetical* changes. If you think "nobody will bother to dot 
everything!" then you are inadvertently arguing that the problem you are 
trying to solve is a tiny problem that most people don't care about.



-- 
Steven



From pyideas at rebertia.com  Tue Apr 26 04:45:20 2011
From: pyideas at rebertia.com (Chris Rebert)
Date: Mon, 25 Apr 2011 19:45:20 -0700
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB62F7A.3050201@pearwood.info>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>
	<4DB5DAB2.6090305@interia.pl> <4DB62F7A.3050201@pearwood.info>
Message-ID: <BANLkTinvaxGvWPnZYFOQh=aQb+UG62eGkA@mail.gmail.com>

On Mon, Apr 25, 2011 at 7:35 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> haael wrote:
>
>> As I said, I want sometimes to import some non-Python namespace, i.e. a
>> Pascal program. If all identifiers are allowed, there would never be a clash
>> of reserved words.
>
> How do you import non-Python code? I don't understand this argument.

I believe Bartosz is referring to bridges to other languages (e.g.
PyObjC), or automatically-generated bindings to non-Python libraries.

Cheers,
Chris


From ncoghlan at gmail.com  Tue Apr 26 05:34:20 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 26 Apr 2011 13:34:20 +1000
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTim87KhxAyhb9V99VNmOuB9K9g9NSw@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>
	<4DB5DAB2.6090305@interia.pl> <4DB5E6ED.40904@stoneleaf.us>
	<BANLkTimLQ_sNdCqEo+Ek5vSJJhXGTssyLQ@mail.gmail.com>
	<BANLkTin3ZTgGoQBSiGfW+28z91gc36KpJA@mail.gmail.com>
	<BANLkTim87KhxAyhb9V99VNmOuB9K9g9NSw@mail.gmail.com>
Message-ID: <BANLkTik1Y64jrr0NnYoyr68U5ZpMZ4qG6g@mail.gmail.com>

On Tue, Apr 26, 2011 at 11:50 AM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
>> You're missing True, False, and None.
>
> Looks like a bug in help("keywords") under Python 3.2.

The bug is actually a little deeper than that - True/False/None were
historically either not keywords at all, or pseudo keywords that were
permitted as attributes, but not as local variables. They haven't yet
been promoted properly to full keyword status in the language grammar
(even though the compiler already ensures that they can't be assigned
to or used as anything other than themselves).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From mwm at mired.org  Tue Apr 26 08:42:00 2011
From: mwm at mired.org (Mike Meyer)
Date: Tue, 26 Apr 2011 02:42:00 -0400
Subject: [Python-ideas] Make all keywords legal as an attribute name
In-Reply-To: <BANLkTinU3LAajpFk0aJUPD6tvwkgnPqH7w@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTimKL6nyP+0Q33BMj209UU-3Z02-=Q@mail.gmail.com>
	<BANLkTike9TLrGW4oxes5-O3qCpcAwXaW9A@mail.gmail.com>
	<BANLkTininhh5B8WCubc_3TRCJLx7=zRjQA@mail.gmail.com>
	<BANLkTinU3LAajpFk0aJUPD6tvwkgnPqH7w@mail.gmail.com>
Message-ID: <20110426024200.653ab68e@bhuda.mired.org>

On Mon, 25 Apr 2011 16:26:47 -0400
Mike Graham <mikegraham at gmail.com> wrote:
> Other examples, some more contrived than others, could be
> provided--some of these would be good names if not for their keyword
> status. However, I don't think I've seen a suggestion better than the
> current solution (or lack thereof).

I think the *first* part of this proposal - allowing attribute names
to be keywords - provides almost all the benefits and few of the
problems that were brought up.

For instance, the "with .with.as as .as" goes away - the worst you can
do is "with with_.as as as_", which is only slightly worse than the
already legal "with with_.as_ as as_".

I don't think it changes the cognitive load of someone reading or
writing code noticeably - limiting it to attributes means you're
limiting it to a namespace that's already visually and logically
distinct from the namespace that keywords can appear in. In fact, that
separation also means there are no backwards compatibility issues.

On the other hand, the problem of wanting to bind external names (say
from a CORBA ORB) should go away, because the names tend to be
attributes of the external objects you are using. The objects
themselves you get to name.

Likewise, most of the examples from the standard library where the
obvious name for something was a keyword were attribute names. This
would fix those, but not let the first parameter to classmethods be
class.

The nasty problem - possibly nasty enough to kill this - are the cases
where context allows bare names to be used to refer to things that are
used as attributes outside the context: a modules global namespace and
class variables off the top of my head. Using the bare name as a
keyword would be a syntax error, but using it as an attribute
elsewhere would work fine. Class variables can probably be worked
around since they only use for bare names is to initialize them at
class definition time, but module global are another issue.

      <mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/consulting.html
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org


From arnodel at gmail.com  Tue Apr 26 09:05:28 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Tue, 26 Apr 2011 08:05:28 +0100
Subject: [Python-ideas] Make all keywords legal as an attribute name
In-Reply-To: <20110426024200.653ab68e@bhuda.mired.org>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTimKL6nyP+0Q33BMj209UU-3Z02-=Q@mail.gmail.com>
	<BANLkTike9TLrGW4oxes5-O3qCpcAwXaW9A@mail.gmail.com>
	<BANLkTininhh5B8WCubc_3TRCJLx7=zRjQA@mail.gmail.com>
	<BANLkTinU3LAajpFk0aJUPD6tvwkgnPqH7w@mail.gmail.com>
	<20110426024200.653ab68e@bhuda.mired.org>
Message-ID: <27596766-E94E-404E-8ED2-7215DE596777@gmail.com>


On 26 Apr 2011, at 07:42, Mike Meyer wrote:

> On Mon, 25 Apr 2011 16:26:47 -0400
> Mike Graham <mikegraham at gmail.com> wrote:
>> Other examples, some more contrived than others, could be
>> provided--some of these would be good names if not for their keyword
>> status. However, I don't think I've seen a suggestion better than the
>> current solution (or lack thereof).
> 
> I think the *first* part of this proposal - allowing attribute names
> to be keywords - provides almost all the benefits and few of the
> problems that were brought up.

Note that attribute names can already be keywords (same for globals).  It's just that the compiler will complain when it sees them, so you have to make sure it doesn't sees them.

>>> class A: pass
... 
>>> a = A()
>>> setattr(a, "with", 1)
>>> getattr(a, "with")
1

>>> globals()["for"] = 12
>>> globals()["for"]
12

-- 
Arnaud



From cs at zip.com.au  Tue Apr 26 09:15:04 2011
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 26 Apr 2011 17:15:04 +1000
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB5E6ED.40904@stoneleaf.us>
References: <4DB5E6ED.40904@stoneleaf.us>
Message-ID: <20110426071504.GA19427@cskk.homeip.net>

On 25Apr2011 14:26, Ethan Furman <ethan at stoneleaf.us> wrote:
| haael wrote:
| >First of all, many nouns are reserved, i.e. "object" or "class".
| 
| Many?  Aren't we still at less than 50 words total?  Pretty
| infinitesimal when compared with the 100,000+ words in the English
| language.
| 
| >Second: variable names are usually nouns indeed, but functions and
| >methods are often verbs, while named parameters can be
| >prepositions and adverbs. [...]
| >Python kidnapped many verbs and prepositions and made them reserved.
| 
| See above.  This is a ridiculous exaggeration.

Though to be fair, Python's using a fair number of the very heavily used
ones.

Personally I'm -1 on the proposal, especially the leading dot part.

One downside that springs to mind is a weakness in C: the syntax is
so... lexically complete... that quite often a syntactic programming
error can get warnings about well below the actual error, and several
easy mistakes are syntacticly valid and only show as logic errors later.
The standard example is = instead of ==.

My point here is that the more valid but mistaken forms the language
allows, the easier it is for simple errors to become logic errors.

Contrived examples:

  # getting the "as" name from "foo"
  from foo import as
  # but did I mean "from foo import this as that" ?

  # is this now a syntax error, since "as" is a name?
  with open("foo") as fp:

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

The worst tyrannies were the ones where a governance required its own logic on
every embedded node. - Vernor Vinge


From stefan_ml at behnel.de  Tue Apr 26 09:15:20 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 26 Apr 2011 09:15:20 +0200
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BANLkTinvaxGvWPnZYFOQh=aQb+UG62eGkA@mail.gmail.com>
References: <4DB5CD47.3060808@interia.pl>	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>	<4DB5DAB2.6090305@interia.pl>
	<4DB62F7A.3050201@pearwood.info>
	<BANLkTinvaxGvWPnZYFOQh=aQb+UG62eGkA@mail.gmail.com>
Message-ID: <ip5re8$e7r$1@dough.gmane.org>

Chris Rebert, 26.04.2011 04:45:
> On Mon, Apr 25, 2011 at 7:35 PM, Steven D'Aprano wrote:
>> haael wrote:
>>
>>> As I said, I want sometimes to import some non-Python namespace, i.e. a
>>> Pascal program. If all identifiers are allowed, there would never be a clash
>>> of reserved words.
>>
>> How do you import non-Python code? I don't understand this argument.
>
> I believe Bartosz is referring to bridges to other languages (e.g.
> PyObjC), or automatically-generated bindings to non-Python libraries.

Those won't be helped much by this proposal, though, given that other 
languages are free to allow or deny as identifiers (and types, objects, 
closures, references, pointers, ...) whatever they like. Wrapping tools 
will always have to be aware of both sides of the wrapper, and deal with 
differences and ambiguities in one way or another. The simple feature that 
"with" could then be used unmangled in Python code does not even touch the 
surface of this.

Stefan



From cs at zip.com.au  Tue Apr 26 09:43:30 2011
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 26 Apr 2011 17:43:30 +1000
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB5DAB2.6090305@interia.pl>
References: <4DB5DAB2.6090305@interia.pl>
Message-ID: <20110426074330.GA25296@cskk.homeip.net>

On 25Apr2011 22:33, haael <haael at interia.pl> wrote:
| @ Mike Graham
| >>Of course, if a keyword is not preceded by a dot, it would be treated as a
| >>reserved word, just like now.
| >>>with = 3  # syntax error
| >I don't see how this is a real improvement over the current
| >convention, to add a trailing underscore, so that programs really
| >needing to use the name "with" would use "with_". [...]
| But the trailing underscore is treated as a part of an identifier,
| while the preceding dot is not. This is important if I want to have
| an identifier named exactly "with", with no other characters (no pun
| itended).
| 
| As I said, I want sometimes to import some non-Python namespace,
| i.e. a Pascal program. If all identifiers are allowed, there would
| never be a clash of reserved words.

Does your proposal help with non-Python namespaces with different
identifier rules? I know this is a little snarky, but I've certainly
seen real world stuff with "$" as a valid identifier character.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

But then, I'm only 50. Things may well get a bit much for me when I
reach the gasping heights of senile decrepitude of which old Andy
Woodward speaks with such feeling.
        - Chris Malcolm, cam at uk.ac.ed.aifh, DoD #205


From masklinn at masklinn.net  Tue Apr 26 10:56:38 2011
From: masklinn at masklinn.net (Masklinn)
Date: Tue, 26 Apr 2011 10:56:38 +0200
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <20110426071504.GA19427@cskk.homeip.net>
References: <4DB5E6ED.40904@stoneleaf.us>
	<20110426071504.GA19427@cskk.homeip.net>
Message-ID: <8639BBF8-927A-4186-8607-DB255471E21C@masklinn.net>

On 2011-04-26, at 09:15 , Cameron Simpson wrote:
> 
> Contrived examples:
> 
>  # getting the "as" name from "foo"
>  from foo import as
>  # but did I mean "from foo import this as that" ?
If you did, you'll realize it quite soon as you'll be missing `that` in your local namespace? In any case, the statement is not ambiguous. Not for the machine, but not for humans either.

> 
>  # is this now a syntax error, since "as" is a name?
>  with open("foo") as fp:
`as` could be used as a name, but would still be a keyword. This statement would be parsed as `with [expr] as [name]`, which is perfectly valid, why would there be a syntax error anywhere?

`with open ("foo") as as` would also be valid, though one could argue less readable.

From ronaldoussoren at mac.com  Tue Apr 26 12:57:26 2011
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 26 Apr 2011 12:57:26 +0200
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <ip5re8$e7r$1@dough.gmane.org>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTinGXLkiVFQb1PHp=O=FKu5TUjnxnQ@mail.gmail.com>
	<4DB5DAB2.6090305@interia.pl> <4DB62F7A.3050201@pearwood.info>
	<BANLkTinvaxGvWPnZYFOQh=aQb+UG62eGkA@mail.gmail.com>
	<ip5re8$e7r$1@dough.gmane.org>
Message-ID: <5B6A0BC0-B508-442E-B27D-C19A91FD3374@mac.com>


On 26 Apr, 2011, at 9:15, Stefan Behnel wrote:

> Chris Rebert, 26.04.2011 04:45:
>> On Mon, Apr 25, 2011 at 7:35 PM, Steven D'Aprano wrote:
>>> haael wrote:
>>> 
>>>> As I said, I want sometimes to import some non-Python namespace, i.e. a
>>>> Pascal program. If all identifiers are allowed, there would never be a clash
>>>> of reserved words.
>>> 
>>> How do you import non-Python code? I don't understand this argument.
>> 
>> I believe Bartosz is referring to bridges to other languages (e.g.
>> PyObjC), or automatically-generated bindings to non-Python libraries.
> 
> Those won't be helped much by this proposal, though, given that other languages are free to allow or deny as identifiers (and types, objects, closures, references, pointers, ...) whatever they like. Wrapping tools will always have to be aware of both sides of the wrapper, and deal with differences and ambiguities in one way or another. The simple feature that "with" could then be used unmangled in Python code does not even touch the surface of this.

They would be helped because the other language might have method/function/class names that are reserved works in Python, an example of this is the 'class' method of NSObject in Objective-C.

PyObjC using the convention of adding a double underscore to the end of method names to make them valid Python identifiers ( anObject.class__()).

I'm -1 on the proposal though, the readability cost is too high for the very small benifit of using keywords as attribute names.

Ronald

> 
> 
> Stefan
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



From cschlick at gmail.com  Tue Apr 26 17:05:21 2011
From: cschlick at gmail.com (Christophe Schlick)
Date: Tue, 26 Apr 2011 17:05:21 +0200
Subject: [Python-ideas] Proposal for new-style decorators
Message-ID: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>

Hello everybody,

My name is Christophe Schlick, from the University of Bordeaux,
France. I've been using Python for years, but it happens that this is
my first post on a Python-related mailing list. For the first time, I
feel that I may have some topic interesting enough for the Python
community, so I would be very happy to get any kind of feeback
(positive or negative) on it. Thanks in advance...

The goal of this post is to propose a new syntax for defining
decorators in Python. I would like to fully explain the rationale and
the benefits of this proposal, as I guess that the discussion might be
more fruitful if all details are written down once for all. However
this has led me to a very long post (a kind of informal PEP) that was
blocked by the moderators. So, following their recommendation, I've
divided the text in three successive parts: (1) the rationale, (2) the
proposal, (3) the proposed implementation and additional
questions/remarks.

For better clarity of the description, I will call the proposed syntax
as "new-style decorators" (NSD, for short) and rename the classic
syntax as "old-style decorators" (OSD), following the terms used some
years ago with the (re)definition of classes. By the way, the
introducing process for NSD shares many aspects with the process used
for introducing new-style classes, including the following features:

* No existing syntax is broken: the only thing required to create a
new-style decorator?is to decorate itself by a newly-introduced
decorator called... "decorator" (well, this sentence is less recursive
than it might appear at the first reading).

* Every thing that can be done with OSD is possible with NSD, but NSD
offer additional more user-friendly features.

* NSD can peacefully live together with OSD in the same code. An NSD
may even decorate an OSD (and vice-versa), however some properties of
the NSD are lost with such a combination.

--------------------------------------------------
1 - Why bother with a new syntax?

To explain what I don't like with the current syntax of decorators,
let me take the example of a basic decorator (called
'old_style_repeat_fix') that simply repeats 3 times its undecorated
function, and adds some tracing to the standard output. Here is the
code:

#---
??def old_style_repeat_fix(func):
?? ?"""docstring for decorating function"""
?? ?# @wraps(func)
?? ?def dummy_func_name_never_used(*args, **keys):
?? ? ?"""docstring for decorated function"""
?? ? ?print "apply 'old_style_repeat_fix' on %r" % func.__name__
?? ? ?for loop in range(3): func(*args, **keys)
?? ?return dummy_func_name_never_used
#---

Even if such code snippets have become quite usual since the
introduction of decorators in Python 2.2, many people have argued (and
I am obviously one of them) that the decorator syntax is a bit
cumbersome. First, it imposes the use of nested functions, which often
reduces readability by moving the function signature and docstring too
far from the corresponding code. Second, as anonymous lambdas
expressions can usually not be employed for decorating functions, the
programmer has no other choice than to create a dummy function name
(only used for one single 'return' statement), which is never a good
coding principle, whatever the programming language. Once you have
tried to teach decorators to a bunch of students, you really
understand how much this syntax leverages the difficulty to grab the
idea.

The situation is even worse when the decorator needs some arguments:
let's create an extended ?decorator (called 'old_style_repeat_var)
that includes an integer 'n' to control the number of iterations, and
a boolean 'trace' to control the tracing behavior. Here is the code:

#---
??def old_style_repeat_var(n=3, trace=True):
?? ?"""docstring for decorating function"""
?? ?def dummy_deco_name_never_used(func):
?? ?"""docstring never used"""
?? ? ?# @wraps(func)
?? ? ?def dummy_func_name_never_used(*args, **keys):
?? ? ? ?"""docstring for decorated function"""
?? ? ? ?if trace:
?? ? ? ? ?print "apply 'old_style_repeat_var' on %r" % func.__name__
?? ? ? ?for loop in range(n): func(*args, **keys)
?? ? ?return dummy_func_name_never_used
?? ?return dummy_deco_name_never_used
#---

This time a two-level function nesting is required and the code needs
two dummy names for these two nested functions. Note that the
docstring of the middle nested function is even totally invisible for
introspection tools. So whether you like nested functions or not,
there is some evidence here that the current syntax is somehow
suboptimal.

Another drawback of OSD is that they do not gently collaborate with
introspection and documentation tools. For instance, let's apply our
decorator on a silly 'test' function:

#---
??@old_style_repeat_var(n=5) # 'trace' keeps its default value
??def test(first=0, last=0):
?? ?"""docstring for undecorated function"""
?? ?print "test: first=%s last=%s" % (first, last)
#---

Now, if we try 'help' on it, we get the following answer:

#---
>>> help(test)
dummy_func_name_never_used(*args, **keys)
?? ?docstring for decorated function
#---

which means that neither the name, nor the docstring, nor the
signature of the 'test' function are correct. Things are a little
better when using the 'wraps' function from the standard 'functools'
module (simply uncomment the line '@wraps(func)' in the code of
'old_style_repeat_var'):

#---
>>> help(test)
test(*args, **keys)
?? ?"""docstring for undecorated function"""
#---

'@wraps(func)' copies the name and the docstring from the undecorated
function to the decorated one, in order to get some useful piece of
information when using 'help'. However, the signature of the function
still comes from the decorated function, not the genuine one.?The
reason is that signature copying is not an easy process. The only
solution is to inspect the undecorated function and then use 'exec' to
generate a wrapper with a correct signature.?This is basically what is
done in the 'decorator' package (available at PyPI) written by Michele
Simionato. There has been a lengthy discussion in python-dev (in 2009
I guess, but I can't find the archive right now) whether to include or
not this package in the standard library.

As far as I know, there is currently no clear consensus whether this
is a good idea or not, because there has always been a mixed feeling
from the community about transparent copy from the undecorated to the
decorated function (even about the 'wraps' function): on one hand,
transparent copy is cool for immediate help, for automatic
documentation and for introspection tools, but on the other hand, it
totally hides the decorating process which is not always what is
wanted... or needed.

The syntax for NSD presented in this proposal tries to improve this
situation by offering two desirable features, according to the Zen of
Python:

*?"flat is better than nested": no nested functions with dummy names
are required, even when parameters are passed to the decorator; only
one single decorating function has to be written by the programmer,
whatever the kind of decorator.

*?"explicit is better than implicit": introspection of a decorated
function explicitely reveals the decoration process, and allows one to
get the name/signature/docstring not only for the corresponding
undecorated function, but also for any number of chained decorators
that have been applied on it.

------
to be continued in Part 2...

CS


From steve at pearwood.info  Tue Apr 26 18:26:39 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 27 Apr 2011 02:26:39 +1000
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
Message-ID: <4DB6F23F.1000006@pearwood.info>

Christophe Schlick wrote:

> The goal of this post is to propose a new syntax for defining
> decorators in Python.

Too long, did read it. Here's the summary so others don't have to:

     "The current syntax for writing decorator functions is
     cumbersome, has too much boilerplate, and doesn't play
     well with introspection tools. I've got some great new
     syntax for writing decorators, but you'll have to wait
     to find out what it is."

You've obviously spent a lot of effort thinking this through, but this 
shouldn't be a mystery novel where you are trying to keep Who Done It 
secret until the end of the book. You talk a lot about new decorator 
syntax, but after reading > 150 lines, I still have no idea what it is, 
how much work it will require, and how disruptive it will be. That 
doesn't make me very enthusiastic about the idea.

Scientific papers and reports often have an executive summary at the 
very beginning: one or two paragraphs that summarize the report without 
all the details. Perhaps you should do the same?

As for the rationale given in this post, I'm not convinced there's 
actually a problem that needs solving. The "problems" you list seem 
pretty minor to me: e.g. so what if you have to name the inner function?

Admittedly, that the function signature is lost when using decorators 
*is* a pretty annoying gotcha, but surely we don't need new *syntax* to 
fix that. New *functionality* in functools.wraps perhaps?

Oh, one thought that comes to me... decorators are a special case of 
factory functions. Does your proposed syntax (whatever it is!) treat 
decorator functions as a special case, or does it apply to general 
factory functions as well?



-- 
Steven


From cschlick at gmail.com  Tue Apr 26 18:52:11 2011
From: cschlick at gmail.com (Christophe Schlick)
Date: Tue, 26 Apr 2011 18:52:11 +0200
Subject: [Python-ideas] Proposal for new-style decorators
Message-ID: <BANLkTimSEy+ErouDRH2PbAaofeZmEx4UDg@mail.gmail.com>

Hi Jim,

The new version. Now cut-and-paste code should work...

------
Hello everybody,

My name is Christophe Schlick, from the University of Bordeaux,
France. I've been using Python for years, but it happens that this is
my first post on a Python-related mailing list. For the first time, I
feel that I may have some topic interesting enough for the Python
community, so I would be very happy to get any kind of feeback
(positive or negative) on it. Thanks in advance...

The goal of this post is to propose a new syntax for defining
decorators in Python. I would like to fully explain the rationale and
the benefits of this proposal, as I guess that the discussion might be
more fruitful if all details are written down once for all. However
this has led me to a very long post (a kind of informal PEP) that was
blocked by the moderators. So, following their recommendation, I've
divided the text in three successive parts: (1) the rationale, (2) the
proposal, (3) the proposed implementation and additional
questions/remarks.

For better clarity of the description, I will call the proposed syntax
as "new-style decorators" (NSD, for short) and rename the classic
syntax as "old-style decorators" (OSD), following the terms used some
years ago with the (re)definition of classes. By the way, the
introducing process for NSD shares many aspects with the process used
for introducing new-style classes, including the following features:

* No existing syntax is broken: the only thing required to create a
new-style decorator is to decorate itself by a newly-introduced
decorator called... "decorator" (well, this sentence is less recursive
than it might appear at the first reading).

* Every thing that can be done with OSD is possible with NSD, but NSD
offer additional more user-friendly features.

* NSD can peacefully live together with OSD in the same code. An NSD
may even decorate an OSD (and vice-versa), however some properties of
the NSD are lost with such a combination.

--------------------------------------------------
1 - Why bother with a new syntax?

To explain what I don't like with the current syntax of decorators,
let me take the example of a basic decorator (called
'old_style_repeat_fix') that simply repeats 3 times its undecorated
function, and adds some tracing to the standard output. Here is the
code:

#---
def old_style_repeat_fix(func):
  """docstring for decorating function"""
  # @wraps(func)
  def dummy_func_name_never_used(*args, **keys):
    """docstring for decorated function"""
    print "apply 'old_style_repeat_fix' on %r" % func.__name__
    for loop in range(3): func(*args, **keys)
  return dummy_func_name_never_used
#---

Even if such code snippets have become quite usual since the
introduction of decorators in Python 2.2, many people have argued (and
I am obviously one of them) that the decorator syntax is a bit
cumbersome. First, it imposes the use of nested functions, which often
reduces readability by moving the function signature and docstring too
far from the corresponding code. Second, as anonymous lambdas
expressions can usually not be employed for decorating functions, the
programmer has no other choice than to create a dummy function name
(only used for one single 'return' statement), which is never a good
coding principle, whatever the programming language. Once you have
tried to teach decorators to a bunch of students, you really
understand how much this syntax leverages the difficulty to grab the
idea.

The situation is even worse when the decorator needs some arguments:
let's create an extended  decorator (called 'old_style_repeat_var)
that includes an integer 'n' to control the number of iterations, and
a boolean 'trace' to control the tracing behavior. Here is the code:

#---
def old_style_repeat_var(n=3, trace=True):
  """docstring for decorating function"""
  def dummy_deco_name_never_used(func):
  """docstring never used"""
    # @wraps(func)
    def dummy_func_name_never_used(*args, **keys):
      """docstring for decorated function"""
      if trace:
        print "apply 'old_style_repeat_var' on %r" % func.__name__
      for loop in range(n): func(*args, **keys)
    return dummy_func_name_never_used
  return dummy_deco_name_never_used
#---

This time a two-level function nesting is required and the code needs
two dummy names for these two nested functions. Note that the
docstring of the middle nested function is even totally invisible for
introspection tools. So whether you like nested functions or not,
there is some evidence here that the current syntax is somehow
suboptimal.

Another drawback of OSD is that they do not gently collaborate with
introspection and documentation tools. For instance, let's apply our
decorator on a silly 'test' function:

#---
@old_style_repeat_var(n=5) # 'trace' keeps its default value
def test(first=0, last=0):
  """docstring for undecorated function"""
  print "test: first=%s last=%s" % (first, last)
#---

Now, if we try 'help' on it, we get the following answer:

#---
>>> help(test)
dummy_func_name_never_used(*args, **keys)
   docstring for decorated function
#---

which means that neither the name, nor the docstring, nor the
signature of the 'test' function are correct. Things are a little
better when using the 'wraps' function from the standard 'functools'
module (simply uncomment the line '@wraps(func)' in the code of
'old_style_repeat_var'):

#---
>>> help(test)
test(*args, **keys)
   """docstring for undecorated function"""
#---

'@wraps(func)' copies the name and the docstring from the undecorated
function to the decorated one, in order to get some useful piece of
information when using 'help'. However, the signature of the function
still comes from the decorated function, not the genuine one. The
reason is that signature copying is not an easy process. The only
solution is to inspect the undecorated function and then use 'exec' to
generate a wrapper with a correct signature. This is basically what is
done in the 'decorator' package (available at PyPI) written by Michele
Simionato. There has been a lengthy discussion in python-dev (in 2009
I guess, but I can't find the archive right now) whether to include or
not this package in the standard library.

As far as I know, there is currently no clear consensus whether this
is a good idea or not, because there has always been a mixed feeling
from the community about transparent copy from the undecorated to the
decorated function (even about the 'wraps' function): on one hand,
transparent copy is cool for immediate help, for automatic
documentation and for introspection tools, but on the other hand, it
totally hides the decorating process which is not always what is
wanted... or needed.

The syntax for NSD presented in this proposal tries to improve this
situation by offering two desirable features, according to the Zen of
Python:

* "flat is better than nested": no nested functions with dummy names
are required, even when parameters are passed to the decorator; only
one single decorating function has to be written by the programmer,
whatever the kind of decorator.

* "explicit is better than implicit": introspection of a decorated
function explicitely reveals the decoration process, and allows one to
get the name/signature/docstring not only for the corresponding
undecorated function, but also for any number of chained decorators
that have been applied on it.

------
to be continued in Part 2...

CS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110426/86bb1def/attachment.html>

From cschlick at gmail.com  Tue Apr 26 19:10:03 2011
From: cschlick at gmail.com (Christophe Schlick)
Date: Tue, 26 Apr 2011 19:10:03 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
Message-ID: <BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>

I am really sorry for splitting the text. My first post was in one
piece but got blocked by the moderators. I didn't want to create some
ridiculous suspense at the end of the first part. Here is the second
part...

---

Part 2 - The new-style syntax for decorators:

Here is the code of the same decorators as in Part 1 above, but the
proposed syntax. The first decorator ('new_style_repeat_fix') is
created without parameters, while the second one
('new_style_repeat_var') uses arguments 'n' and 'trace' with the same
role as previously:

#---
@decorator
def new_style_repeat_fix(self, *args, **keys):
  """docstring for decorating function"""
  print "apply %r on %r" % (self.deco.__name__, self.func.__name__)
  for n in range(3): self.func(*args, **keys)

@decorator(n=3, trace=True)
def new_style_repeat_var(self, *args, **keys):
  """docstring for decorating function"""
  if self.trace:
    print "apply %r on %r" % (self.deco.__name__, self.func.__name__)
  for n in range(self.n): self.func(*args, **keys)
#---

When examining the new-style syntax, one can notice that there are
basically four characteristics that distinguish NSD from OSD:

* Each NSD is decorated by a newly-introduced callable class called
'decorator' (code provided in Part 3) by using one of two possible
decorating forms. The first form '@decorator' is employed for
decorators that do not need any parameter. The second form
'@decorator(arg1=val1, arg2=val2...)' is employed to specify a
sequence of named arguments (combined with their default values) that
are passed to the decorator using the standard notation for keyword
arguments.

* The role of the 'decorator' class is to generate the decorated
function (i.e. the inner nested function with the classic OSD syntax)
and to broadcast it to the decorating function as its first argument
'self'. When the second decorating form is used, all keyword arguments
used in '@decorator(...)' are automatically injected as
meta-attributes in this decorated function 'self'. In the example
above, the two decorator arguments 'n' and 'trace' are available
within the code of 'new_style_repeat_var' as 'self.n' and 'self.trace'
respectively. This mechanism avoids one level of nested functions used
in standard OSD.

* In addition to these decorator arguments, the decorating process
also injects two other meta-attributes in the decorated function:
'self.deco' represents a reference to the decorating function, while
'self.func' represents a reference to the undecorated function. If
there are several chained decorators, the mechanism is made recursive
(more on this later). Note that this implies a slight name
restriction: neither 'deco' nor 'func' can be used as the name of a
parameter passed to the decorator, as this would generate collision in
the corresponding namespace. An alternative might be to consider the
two references as "special" attributes and rename them as
'self.__deco__' and 'self.__func__' respectively. I have no clear
opinion about the pros/cons of the two alternatives.

* Finally, note that each NSD has the same 3-argument signature:
(self, *args, **keys). The first argument 'self' has been explained
above. The two others 'args' and 'keys' respectively represent the set
of positional and keyword arguments, as usual. However, all the values
in either 'args' or 'keys' are not meant to be used by the decorating
function, but always directly passed to the undecorated function. This
means that the statement 'self.func(*args, **keys)' will always appear
somewhere in the code of an NSD. Following this mechanism in the
decorating function avoids the other level of nested functions used in
standard OSD, and guarantees that flat functions are always
sufficient.

Once the NSD have been defined with the new syntax, they can be used
to decorate functions using the standard @-notation, either for single
or multiple decoration. For instance:

#---
@new_style_repeat_fix
def testA(first=0, last=0):
  """docstring for undecorated function"""
  print "testA: first=%s last=%s" % (first, last)

@new_style_repeat_var(n=5) # 'n' is changed, 'trace' keeps default value
def testB(first=0, last=0):
  """docstring for undecorated function"""
  print "testB: first=%s last=%s" % (first, last)

@new_style_repeat_var # both 'n' and 'trace' keep their default values
@new_style_repeat_fix
@new_style_repeat_var(n=5, trace=False) # both 'n' and 'trace' are changed
def testC(first=0, last=0):
  """docstring for undecorated function"""
  print "testC: first=%s last=%s" % (first, last)
#---

When applying a decorator without arguments, or when *all* its
arguments use their default values, the parenthesis after the
decorator name may be dropped. In other words, '@mydeco' and
'@mydeco()' are totally equivalent, whether 'mydeco' takes arguments
or not. This solves a non-symmetric behavior of standard OSD that has
always bothered me: '@old_style_repeat_fix' works but
'@old_style_repeat_fix()' does not, and inversely
'@old_style_repeat_var()' works but '@old_style_repeat_var' does not.

Note also that in the case of chained decorators, each decoration
level stores its own set of parameters, so there is no conflict when
applying the same decorator several times on the same function, as
done with 'new_style_repeat_var' on 'testC'.

Now let's play a bit with some introspection tools:

#---
>>> testA
<function <deco>testA...>

>>> testB
<function <deco>testB...>

>>> testC
<function <deco><deco><deco>testC...>
#---

To explicitely expose the decoration process, a '<deco>' substring is
added as a prefix to the '__name__' attribute for each decorated
function (more precisely, there is one '<deco>' for each level of
decoration, as can be seen with 'testC'). So, each time a '<deco>'
prefix is encountered, the user knows that the reference to the
corresponding undecorated function (resp. decorating function) is
available through the meta-attribute '.func' (resp. '.deco'). When
calling 'help' on a decorated function, this principle is clearly
displayed, and the user can thus easily obtain useful information,
including correct name/signature/docstring:

#---
>>> help(testA)
<deco>testA(*args, **keys)
    use help(testA.func) to get genuine help

>>> testA.func, testA.deco
(<function testA...>, <function new_style_repeat_fix...>)

>>> help(testA.func)
testA(first=0, last=0)
    docstring for undecorated function

>>> help(testA.deco)
new_style_repeat_fix(self, *args, **keys)
    docstring for decorating function
#---

In the case of chained decorators, the same principle holds
recursively. As can be seen in the example below, all information
relative to a multi-decorated function (including all decorator
arguments used at any decoration level) can be easily fetched by
successive applications of the '.func' suffix:

#---
>>> help(testC)
<deco><deco><deco>testC(*args, **keys)
    use help(testC.func.func.func) to get genuine help

>>> testC.func, testC.deco, testC.n, testC.trace
(<function <deco><deco>testC...>, <function new_style_repeat_var...>, 3, True)

>>> testC.func.func, testC.func.deco
(<function <deco>testC...>, <function new_style_repeat_fix...>)

>>> testC.func.func.func, testC.func.func.deco, testC.func.func.n, testC.func.func.trace
(<function testC...>, <function new_style_repeat_var...>, 5, False)

>>> help(testC.func.func.func)
testC(first=0, last=0)
    docstring for undecorated function
#---

------
to be continued in Part 3...

CS


From cschlick at gmail.com  Tue Apr 26 19:11:23 2011
From: cschlick at gmail.com (Christophe Schlick)
Date: Tue, 26 Apr 2011 19:11:23 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
Message-ID: <BANLkTinLSHnS4ef6AHcLs2S_XnGfgfT_UA@mail.gmail.com>

Part 3 - Implementation and additional remarks/questions

I've implemented the idea of NSD about nine months ago and have used
them successfully in many different situations. My first
implementation was terribly messy but at least it did the job. Last
week, while thinking about refactoring, I realized that the whole
process can be easily implemented as a state machine. This has led me
to an extremely compact implementation (about 30 Python statements),
where a callable class repeatedly returns reference to itself, until
it gets all the required parameters to generate and return the
decorated function.
Here is the code, so that you can experiment with it, if you wish:

#---
class decorator(object):
  """apply a 'new-style' decorator to a function"""
  def __init__(self, deco=None, **keys):
    # either get reference or default parameters for decorating function
    self.deco = deco; self.keys = keys; self.stack = list()
  def __call__(self, func=None, **keys):
    if self.deco is None: # get reference for decorating function
      self.deco = func; return self
    elif func is None: # get modified parameters of decorating function
      self.stack.append(keys); return self
    else: # get undecorated function and generate decorated function
      deco = node = lambda *args, **keys: self.deco(deco, *args, **keys)
      deco.func = func; deco.deco = self.deco; deco.__dict__.update(self.keys)
      if self.stack: deco.__dict__.update(self.stack.pop())
      head = '<deco>'; deco.__name__ = name = head + func.__name__
      level = name.count(head); offset = len(head)*level; tail = '.func'*level
      doc = "use help(%s) to get genuine help" % (name[offset:] + tail)
      while hasattr(node, 'func'): node.__doc__ = doc; node = node.func
      return deco
#---

The simplicity of the new implementation has convinced me that it
might be useful to share this idea and write a proposal in order to
get some feedback from the community. As said in the introduction,
this is my first post to python-ideas, so I'm not sure about the
correct process to follow. I've got plenty of questions anyway:

* Is the idea interesting enough to deserve consideration for possible
inclusion in the language? If yes, should I transform this proposal
into a PEP, or should there first be some pre-PEP discussion here (or
maybe in python-dev)?

* Are there some pitfalls involved with the use of NSD that I haven't
seen? Or are there additional desirable elements that could be easily
included?

* After having read this proposal, has anybody some suggestion for
alternative syntax that offer similar features?

* There are some advanced features offered by the new syntax (such as
meta-decorator, or whatever you call them), which seem to be powerful
but are less stable than the elements presented here. I did not detail
this kind of stuff because I consider that it is likely to create some
noise in the discussion, but maybe not?

Thanks for any opinion,
CS


From stutzbach at google.com  Tue Apr 26 19:42:23 2011
From: stutzbach at google.com (Daniel Stutzbach)
Date: Tue, 26 Apr 2011 10:42:23 -0700
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
Message-ID: <BANLkTi==2Bx5Hu5_25MhxN3fNG8v11Y8RA@mail.gmail.com>

On Tue, Apr 26, 2011 at 10:10 AM, Christophe Schlick <cschlick at gmail.com>wrote:

> @decorator
> def new_style_repeat_fix(self, *args, **keys):
>  """docstring for decorating function"""
>  print "apply %r on %r" % (self.deco.__name__, self.func.__name__)
>  for n in range(3): self.func(*args, **keys)


I'm sure it's not exactly as you envisioned, but the decorator package
provides roughly what you're describing:

http://micheles.googlecode.com/hg/decorator/documentation.html#decorator-is-a-decorator

-- 
Daniel Stutzbach
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110426/e6ebf618/attachment.html>

From ethan at stoneleaf.us  Tue Apr 26 19:58:31 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 26 Apr 2011 10:58:31 -0700
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTi==2Bx5Hu5_25MhxN3fNG8v11Y8RA@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>	<BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
	<BANLkTi==2Bx5Hu5_25MhxN3fNG8v11Y8RA@mail.gmail.com>
Message-ID: <4DB707C7.7040502@stoneleaf.us>

Daniel Stutzbach wrote:
> On Tue, Apr 26, 2011 at 10:10 AM, Christophe Schlick <cschlick at gmail.com 
> <mailto:cschlick at gmail.com>> wrote:
> 
>     @decorator
>     def new_style_repeat_fix(self, *args, **keys):
>      """docstring for decorating function"""
>      print "apply %r on %r" % (self.deco.__name__, self.func.__name__)
>      for n in range(3): self.func(*args, **keys)
> 
> 
> I'm sure it's not exactly as you envisioned, but the decorator package 
> provides roughly what you're describing:
> 
> http://micheles.googlecode.com/hg/decorator/documentation.html#decorator-is-a-decorator
> 

I believe he mentioned Michele's code in his first post.

I like the extra introspection Christophe's code offers.

Was there any syntax change beyond

    @decorator

and

    @decorator()

behaving the same?

~Ethan~


From steve at pearwood.info  Tue Apr 26 20:08:18 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 27 Apr 2011 04:08:18 +1000
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTinLSHnS4ef6AHcLs2S_XnGfgfT_UA@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>	<BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
	<BANLkTinLSHnS4ef6AHcLs2S_XnGfgfT_UA@mail.gmail.com>
Message-ID: <4DB70A12.9030601@pearwood.info>

Christophe Schlick wrote:

> * Is the idea interesting enough to deserve consideration for possible
> inclusion in the language? If yes, should I transform this proposal
> into a PEP, or should there first be some pre-PEP discussion here (or
> maybe in python-dev)?

Decorators were one of the biggest successes in recent years, so we 
would be foolish to dismiss the idea of simplifying them out of hand. 
But I think you are doing yourself a disservice by referring to this 
proposal as "new syntax". Normally when people talk about syntax, they 
are referring to language syntax (i.e. a change to the Python 
interpreter), and we're pretty conservative about adding new syntax.

It seems to me that you're talking about a new idiom for building 
decorator functions, not new syntax.

I would suggest you also publish this decorator-builder recipe on 
ActiveState's Python cookbook, and see if you get much interest there. 
It might also help to post a link to your recipe to 
python-list at python.org. You certainly should do those things before 
going to python-dev.



> * Are there some pitfalls involved with the use of NSD that I haven't
> seen? Or are there additional desirable elements that could be easily
> included?


Have you timed the decorated function using new and old style? If you 
decorator a function with (say) 5 arguments, is there any performance 
hit to your NSD?

Do you have any tests for it? E.g. unit tests, regression tests? Your 
code looks opaque and complicated to me, I would want to see a good test 
suite before even considering using it in production code, let alone in 
the standard library.




-- 
Steven



From cschlick at gmail.com  Tue Apr 26 20:45:57 2011
From: cschlick at gmail.com (Christophe Schlick)
Date: Tue, 26 Apr 2011 20:45:57 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <4DB70A12.9030601@pearwood.info>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
	<BANLkTinLSHnS4ef6AHcLs2S_XnGfgfT_UA@mail.gmail.com>
	<4DB70A12.9030601@pearwood.info>
Message-ID: <BANLkTimh4jhSVyh1d3KBmeTucQm3obUkjA@mail.gmail.com>

On Tue, Apr 26, 2011 at 8:08 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Decorators were one of the biggest successes in recent years, so we would be
> foolish to dismiss the idea of simplifying them out of hand. But I think you
> are doing yourself a disservice by referring to this proposal as "new
> syntax". Normally when people talk about syntax, they are referring to
> language syntax (i.e. a change to the Python interpreter), and we're pretty
> conservative about adding new syntax.

You are right. I've mostly used the expression "new-style decorators"
(which corresponds to the idea of "new idiom" that you propose) but I
guess that there are still a couple of unwanted "new syntax" in the
post. At the beginning of the post, I say that "no existing syntax is
broken", so the proposal does not change anything for the interpreter.
Moreover, the old idiom can be used in combination with the new one,
so no existing code can be broken by the proposal.

CS


From tjreedy at udel.edu  Tue Apr 26 23:20:38 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 26 Apr 2011 17:20:38 -0400
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
Message-ID: <ip7cv9$tck$1@dough.gmane.org>

On 4/26/2011 11:05 AM, Christophe Schlick wrote:

I got interrupted in responding this, and you have now posted parts 2 
and 3, so I will snip and revise a bit.

 > a new syntax for defining decorators in Python.

There is no special syntax for defining decorators -- just normal nested 
function or class definition syntax. To put it another way, Python does 
not have decorator objects. A decorator is simply a callable (function, 
class, or class instance with __call__ method) applied to a object 
(function or class) with the @deco syntax (before the def/class 
statement) instead of the normal call syntax (after the def/class 
statement). Decorators return either the original object (usually 
modified) or another object that is usually, but not necessarily, of the 
same kind as the input.

As Stephen noted, syntax is what is defined in the Language reference. 
Code patterns are what are embodied in the stdlib (or pypi or the Python 
cookbook or other example repositories).

What you are actually proposing is a meta-decorator (class) whose 
instances can be used as decorators because the class has a __call__ 
instance method. This sort of thing is a known alternative to the nested 
function pattern.

> programmer has no other choice than to create a dummy function name

Many of us consider dummy names a bogus problem. I recommend you skip 
this argument. In any case, this, like the other 'problems' you describe 
for nested functions, has nothing in particular with their usage as 
decorators.

> (only used for one single 'return' statement), which is never a good
> coding principle, whatever the programming language.

This strikes me as a bogus rule: a single use of a local name is quite 
common, and not just in Python. I recommend leaving this also out of 
your problem list. Stick with the two real problems.

1. The double or triple nested function pattern has a lot of boilerplate 
and can be difficult to learn. Hiding boilerplate in a class makes the 
use pattern simpler and easier to learn. This is a real benefit. One of 
the three major benefits of writing a generator function versus an 
equivalent iterator class is that is hides the boilerplate required for 
the latter. Similarly, for loops hide the boilerplate required for an 
equivalent while loop.

2. Introspection (more comments below), which your class also addresses.

> #---
>    def old_style_repeat_var(n=3, trace=True):
>      """docstring for decorating function"""

Actually, this is the docstring for the decorator making function.

>      def dummy_deco_name_never_used(func):
>      """docstring never used"""
>        # @wraps(func)
>        def dummy_func_name_never_used(*args, **keys):
>          """docstring for decorated function"""
>          if trace:
>            print "apply 'old_style_repeat_var' on %r" % func.__name__
>          for loop in range(n): func(*args, **keys)
>        return dummy_func_name_never_used
>      return dummy_deco_name_never_used
> #---

> This time a two-level function nesting is required and the code needs
> two dummy names for these two nested functions.

'deco' and 'wrapper' work for me. But I agree that this is a bit 
confusing. But I think that is somewhat inherent in calling a 
decorator-maker f1 to return decorator f2 that returns wrapper f3 that 
wraps the original function f.

 > Note that the  docstring of the middle nested function
 > is even totally invisible for introspection tools.

Not true. To see the docstring of a dynamically created temporary 
function, you have to either dynamically create it or dig inside the 
function that creates it to find the constant string:

 >>> old_style_repeat_var().__doc__
'docstring never used'

 >>> old_style_repeat_var.__code__.co_consts[1].co_consts[0]
'docstring never used'

But I am not sure why you would want to see it or even have one.

> So whether you like nested functions or not,
> there is some evidence here that the current syntax is somehow
> suboptimal.

The 'problems' of nested defs has nothing to do with decorators in 
particular. Functional programmers use them all the time.

> Another drawback of OSD is that they do not gently collaborate with
> introspection and documentation tools. For instance, let's apply our
> decorator on a silly 'test' function:
>
> #---
>    @old_style_repeat_var(n=5) # 'trace' keeps its default value
>    def test(first=0, last=0):
>      """docstring for undecorated function"""
>      print "test: first=%s last=%s" % (first, last)
> #---
>
> Now, if we try 'help' on it, we get the following answer:
>
> #---
>>>> help(test)
> dummy_func_name_never_used(*args, **keys)
>      docstring for decorated function
> #---

Only because you commented out @wraps.
Again, this is not a problem of the @decorator syntax but of *all* 
wrapping callables. Functools.partial has the same 'problem'.

> '@wraps(func)' copies the name and the docstring from the undecorated
> function to the decorated one, in order to get some useful piece of
> information when using 'help'. However, the signature of the function
> still comes from the decorated function, not the genuine one.

I am not sure what you mean. If the two signatures are different, then 
one must use the signature of the wrapper when calling it, not the 
signature of the wrappee, which is perhaps what you mean by 'the genuine
one'. The problem of generic wrappers having generic signatures (*args, 
**kwds) is endemic to using generic wrappers instead of special case 
wrappers.

> reason is that signature copying is not an easy process.

True if you want to do it generically. Copying with modification, as 
functools.partial would have to do, is even worse.

> The only
> solution is to inspect the undecorated function and then use 'exec' to
> generate a wrapper with a correct signature. This is basically what is
> done in the 'decorator' package (available at PyPI) written by Michele
> Simionato. There has been a lengthy discussion in python-dev (in 2009
> I guess, but I can't find the archive right now) whether to include or
> not this package in the standard library.

The other solution is to not use a generic wrappers with generic 
signatures but to write specific wrappers with the actual signature, 
which people did, for instance, before functools and partial() were 
added to Python.

There have been proposals but no consensus on a decorator or decolib 
module for the stdlib. I second the other recommendations to make your 
proposal available on the cookbook site, etc.

-- 
Terry Jan Reedy



From pyideas at rebertia.com  Wed Apr 27 00:30:48 2011
From: pyideas at rebertia.com (Chris Rebert)
Date: Tue, 26 Apr 2011 15:30:48 -0700
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
Message-ID: <BANLkTinuJEtvk53DycF7Dh-vAoBCFOUFog@mail.gmail.com>

On Tue, Apr 26, 2011 at 10:10 AM, Christophe Schlick <cschlick at gmail.com> wrote:
<snip>
> Part 2 - The new-style syntax for decorators:
>
> Here is the code of the same decorators as in Part 1 above, but the
> proposed syntax. The first decorator ('new_style_repeat_fix') is
> created without parameters, while the second one
> ('new_style_repeat_var') uses arguments 'n' and 'trace' with the same
> role as previously:
>
> #---
> @decorator
> def new_style_repeat_fix(self, *args, **keys):
> ?"""docstring for decorating function"""
> ?print "apply %r on %r" % (self.deco.__name__, self.func.__name__)
> ?for n in range(3): self.func(*args, **keys)
>
> @decorator(n=3, trace=True)
> def new_style_repeat_var(self, *args, **keys):
> ?"""docstring for decorating function"""
> ?if self.trace:
> ? ?print "apply %r on %r" % (self.deco.__name__, self.func.__name__)
> ?for n in range(self.n): self.func(*args, **keys)

I'm personally not a fan of magic such as having both @decorator and
@decorator(...) work.

> When examining the new-style syntax, one can notice that there are
> basically four characteristics that distinguish NSD from OSD:
>
> * Each NSD is decorated by a newly-introduced callable class called
> 'decorator' (code provided in Part 3) by using one of two possible
> decorating forms. The first form '@decorator' is employed for
> decorators that do not need any parameter. The second form
> '@decorator(arg1=val1, arg2=val2...)' is employed to specify a
> sequence of named arguments (combined with their default values) that
> are passed to the decorator using the standard notation for keyword
> arguments.

What if my decorator's parameters don't all have default values?
What if I don't want people to have to use keyword arguments when
using my decorator?
What if my decorator accepts extra positional arguments (i.e. *args)?

Cheers,
Chris


From pyideas at rebertia.com  Wed Apr 27 00:39:16 2011
From: pyideas at rebertia.com (Chris Rebert)
Date: Tue, 26 Apr 2011 15:39:16 -0700
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTinLSHnS4ef6AHcLs2S_XnGfgfT_UA@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
	<BANLkTinLSHnS4ef6AHcLs2S_XnGfgfT_UA@mail.gmail.com>
Message-ID: <BANLkTimNzXnFOWh9V24nj9Ex0jdP6JuM3A@mail.gmail.com>

On Tue, Apr 26, 2011 at 10:11 AM, Christophe Schlick <cschlick at gmail.com> wrote:
> Part 3 - Implementation and additional remarks/questions
<snip>
> Here is the code, so that you can experiment with it, if you wish:
>
> #---
> class decorator(object):
> ?"""apply a 'new-style' decorator to a function"""
> ?def __init__(self, deco=None, **keys):
> ? ?# either get reference or default parameters for decorating function
> ? ?self.deco = deco; self.keys = keys; self.stack = list()
> ?def __call__(self, func=None, **keys):

(Somewhat contrived): What if my decorator or the decorate-ee has a
parameter named `deco` or `func`?

Cheers,
Chris


From cschlick at gmail.com  Wed Apr 27 03:42:22 2011
From: cschlick at gmail.com (Christophe Schlick)
Date: Wed, 27 Apr 2011 03:42:22 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTimw85KyORHXwSJHuuFH-_j_c=R1TQ@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<ip7cv9$tck$1@dough.gmane.org>
	<BANLkTimw85KyORHXwSJHuuFH-_j_c=R1TQ@mail.gmail.com>
Message-ID: <BANLkTim_z7SMV_zf_tvATE9QjbhjmRZjBw@mail.gmail.com>

On Tue, Apr 26, 2011 at 11:20 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>
> There is no special syntax for defining decorators -- just normal nested
> function or class definition syntax. To put it another way, Python does not
> have decorator objects. A decorator is simply a callable (function, class,
> or class instance with __call__ method) applied to a object

 I totally agree with that. When defining a decorating function, you
 don't have any syntactic element that could explain the reader of your
 code that this function is actually a decorator. It is only when
 applied on a function with the @-syntax that the mechanism becomes
 visible (but this is not always done in the same file). This can be
 considered as a strength (any function with a correct input/output is
 likely to later become a decorator, even if the original author did
 not thought about it). However, according to my own little experience,
 more than 9 times out of 10, you perfectly now when writting such a
 function that it is actually a decorator. Using the proposed
 @decorator idiom (hey see, I haven't written "syntax" ;-) has at least
 the advantage to be explicit when you want to be explicit (besides the
 other features it provides),

> What you are actually proposing is a meta-decorator (class) whose instances
> can be used as decorators because the class has a __call__ instance method.
> This sort of thing is a known alternative to the nested function pattern.

 Yes, I know that using callable class can be an alternative to the
 nested function pattern. In the pattern you talk about, the __init__
 method gets the decorator arguments, the __call__ method serves as a
 decorator making function, and a third method is used to generate the
 actual decorating function. As a result, the boilerplate is
 approximatively the same as with the nested functions idiom. But this
 is not what is done here, because the end-user only writes a single
 function, not a whole class.

 In my proposal, the two nested functions are avoided by the fact that
 (1) the decorator attributes are automatically injected as
 meta-attributes (this is the role of the middle nested function in the
 standard idiom), and (2) the decorating function is in charge to pass
 the whole set of arguments to the undecorated function (this is the
 role of the inner nested function in the standard idiom). As far as I
 know, I haven't seen the combination of these two elements before.

> ?Stick with the two real problems.
>
> 1. The double or triple nested function pattern has a lot of boilerplate and
> can be difficult to learn. Hiding boilerplate in a class makes the use
> pattern simpler and easier to learn. This is a real benefit.

> 2. Introspection (more comments below), which your class also addresses.

 OK. I'll drop the arguments on nested function, and simply focus on
 boilerplate and introspection. That makes sense.

>> ? def old_style_repeat_var(n=3, trace=True):
>> ? ? """docstring for decorating function"""
>
> Actually, this is the docstring for the decorator making function.

 Also agree. I've written "decorating function" by symmetry with the
 new idiom, but I knew that I would get some remark here ;-)

>> '@wraps(func)' copies the name and the docstring from the undecorated
>> function to the decorated one, in order to get some useful piece of
>> information when using 'help'. However, the signature of the function
>> still comes from the decorated function, not the genuine one.
>
> I am not sure what you mean. If the two signatures are different, then one
> must use the signature of the wrapper when calling it, not the signature of
> the wrappee, which is perhaps what you mean by 'the genuine
> one'. The problem of generic wrappers having generic signatures (*args,
> **kwds) is endemic to using generic wrappers instead of special case
> wrappers.

 What I wanted to say is that wraps only does half of the job: it
 correctly copies the name and the docstring, but the signature
 presented by the help function is still test(*args, **keys), while it
 should actually be test(first=0, last=0) according to the undecorated
 function.

 The alternative proposed by the new idiom is to copy nothing at all:
 it simply says "OK, 'test' is a decorated function. If you want to
 know more look at 'test.func' to get the info about the undecorated
 one, and at 'test.deco' to see what the decorator has done". I prefer
 such a raw-but-explicit approach rather than an automatic half-baked,
 half-bogus one. Moreover, it is not easy with the 'functools' module
 to provide introspection of the decorating function, once you've got
 the decorated one.

>> The only
>> solution is to inspect the undecorated function and then use 'exec' to
>> generate a wrapper with a correct signature. This is basically what is
>> done in the 'decorator' package (available at PyPI) written by Michele
>> Simionato. There has been a lengthy discussion in python-dev (in 2009
>> I guess, but I can't find the archive right now) whether to include or
>> not this package in the standard library.
>
> The other solution is to not use a generic wrappers with generic signatures
> but to write specific wrappers with the actual signature, which people did,
> for instance, before functools and partial() were added to Python.

 Right again, but this overweights the boilerplate even more compared
 to the '@wraps' decorator, no?

> I second the other recommendations to make your proposal
> available on the cookbook site, etc.

 That sounds good... Thanks a lot, Terry!


From cschlick at gmail.com  Wed Apr 27 03:58:24 2011
From: cschlick at gmail.com (Christophe Schlick)
Date: Wed, 27 Apr 2011 03:58:24 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTinuJEtvk53DycF7Dh-vAoBCFOUFog@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
	<BANLkTinuJEtvk53DycF7Dh-vAoBCFOUFog@mail.gmail.com>
Message-ID: <BANLkTimoQ8WWCqAJM5iON0jpUgYZbMC8ug@mail.gmail.com>

On Wed, Apr 27, 2011 at 12:30 AM, Chris Rebert <pyideas at rebertia.com> wrote:
> On Tue, Apr 26, 2011 at 10:10 AM, Christophe Schlick <cschlick at gmail.com> wrote:
>> @decorator
>> ...
>> @decorator(n=3, trace=True)
>>...
>
> I'm personally not a fan of magic such as having both @decorator and
> @decorator(...) work.

Well, you may simply use @decorator(), if your decorator does not need
arguments. Actually the magic does not come from the new 'decorator'
class, but directly from the state machine used to parse the @-syntax
in Python. I am not responsible for that.

> What if my decorator's parameters don't all have default values?
> What if I don't want people to have to use keyword arguments when
> using my decorator?
> What if my decorator accepts extra positional arguments (i.e. *args)?

The idea behind the proposal is to reduce the boilerplate for most of
the standard patterns of decorators. One element of that reduction is
to automatically transform the decorator arguments as attributes of
the decorated function. To do this, the attributes have to get
individual names, that's why I've proposed the keyword argument
syntax. However, it is actually possible to implement the same idea by
letting the decorator use positional arguments, which are then
combined into a single tuple attribute 'self.args' available for the
decorating function. The code of the 'decorator' class would simply be
a bit longer, but there is no specific difficulty here.

 CS


From cschlick at gmail.com  Wed Apr 27 04:24:32 2011
From: cschlick at gmail.com (Christophe Schlick)
Date: Wed, 27 Apr 2011 04:24:32 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <4DB70A12.9030601@pearwood.info>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
	<BANLkTinLSHnS4ef6AHcLs2S_XnGfgfT_UA@mail.gmail.com>
	<4DB70A12.9030601@pearwood.info>
Message-ID: <BANLkTinaLOB0=mKrf0BuVZfXexrGcPRu4w@mail.gmail.com>

On Tue, Apr 26, 2011 at 8:08 PM, Steven D'Aprano <steve at pearwood.info> wrote:

> I would suggest you also publish this decorator-builder recipe on
> ActiveState's Python cookbook, and see if you get much interest there. It
> might also help to post a link to your recipe to python-list at python.org. You
> certainly should do those things before going to python-dev.

OK, I'm going to try that. Thanks.
>
>> * Are there some pitfalls involved with the use of NSD that I haven't
>> seen? Or are there additional desirable elements that could be easily
>> included?
>
> Have you timed the decorated function using new and old style? If you
> decorator a function with (say) 5 arguments, is there any performance hit to
> your NSD?

Intuitively I would say the the only performance hit would comme from
the fact that the decorator arguments are accessed via self.__dict__
in NSD, while there are available as locals with OSD. I've made some
quick 'timeit' tests. I don't know if this is the kind of timing you
thought about:

#---
from timeit import Timer
from decorator import decorator

# OSD
def old_add_args(a=1, b=2, c=3, d=4, e=5):
  def dummy1(func):
    def dummy2(*args, **keys):
      return a + b + c + d + e + func(*args, **keys)
    return dummy2
  return dummy1

# NSD
@decorator(a=1, b=2, c=3, d=4, e=5)
def new_add_args(self, *args, **keys):
  return self.a + self.b + self.c + self.d + self.e + self.func(*args, **keys)

# Apply OSD
@old_add_args()
def old_test(*args, **keys):
  return sum(*args)

# Apply NSD
@new_add_args()
def new_test(*args, **keys):
  return sum(*args)

# Gentle case: the evaluation of the function is rather long compared
to the time
# needed to fetch the 5 decorator args
old_time = Timer('old_test(range(999))', 'from __main__ import
old_test').timeit()
new_time = Timer('new_test(range(999))', 'from __main__ import
new_test').timeit()
print "Gentle: old = %.3f new = %.3f" % (old_time, new_time)

# Worst case: the evaluation of the function is negligible compared to the time
# needed to get the 5 decorators args.
old_time = Timer('old_test(range(1))', 'from __main__ import old_test').timeit()
new_time = Timer('new_test(range(1))', 'from __main__ import new_test').timeit()
print "Worst: old = %.3f new = %.3f" % (old_time, new_time)
#---

Here are the timings obtained on my notebook:

Gentle: old = 45.983 new = 46.377
Worst: old = 4.043 new = 5.127

which seems to confirm that the overhead mainly comes from the
'self.xxx' fetch, and it pretty negligible when heavy computation is
performed.

CS


From stephen at xemacs.org  Wed Apr 27 04:44:28 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 27 Apr 2011 11:44:28 +0900
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTim_z7SMV_zf_tvATE9QjbhjmRZjBw@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<ip7cv9$tck$1@dough.gmane.org>
	<BANLkTimw85KyORHXwSJHuuFH-_j_c=R1TQ@mail.gmail.com>
	<BANLkTim_z7SMV_zf_tvATE9QjbhjmRZjBw@mail.gmail.com>
Message-ID: <87aafc9zw3.fsf@uwakimon.sk.tsukuba.ac.jp>

Christophe Schlick writes:

 >  What I wanted to say is that wraps only does half of the job: it
 >  correctly copies the name and the docstring, but the signature
 >  presented by the help function is still test(*args, **keys), while it
 >  should actually be test(first=0, last=0) according to the undecorated
 >  function.

But that's a lie; the undecorated function is *not* what *my* code
calls.  I would be very confused if I committed a syntax error
according to the help, but the compiler let it go silently.  In your
approach, this doesn't get caught until the erroneous call is actually
made, which might be after the code is put into production.  Or it may
not get caught at all, depending on whether the
decorater-decorater-decorated function checks arguments or simply
swallows unneeded arguments.

In the standard approach, I see "test(*args, **keys)" and go "!@#$%
Name your !@#$% arguments, for heaven's sake!" and go read the code
(the first time), or "oh, it's decorated, gotta read the code, I guess
... mmrmfrmblgrr" (with experience).  But the help's less-than-
informative signature tells me I need to review carefully, whereas in
your approach I would tend to leave it up to the compiler to some
extent.


From cschlick at gmail.com  Wed Apr 27 05:02:07 2011
From: cschlick at gmail.com (Christophe Schlick)
Date: Wed, 27 Apr 2011 05:02:07 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <87aafc9zw3.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<ip7cv9$tck$1@dough.gmane.org>
	<BANLkTimw85KyORHXwSJHuuFH-_j_c=R1TQ@mail.gmail.com>
	<BANLkTim_z7SMV_zf_tvATE9QjbhjmRZjBw@mail.gmail.com>
	<87aafc9zw3.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <BANLkTi=ULc_4Hfb97cKm7Ae_HNFiWYLnXw@mail.gmail.com>

On Wed, Apr 27, 2011 at 4:44 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:

> In the standard approach, I see "test(*args, **keys)" and go "!@#$%
> Name your !@#$% arguments, for heaven's sake!" and go read the code
> (the first time), or "oh, it's decorated, gotta read the code, I guess
> ... mmrmfrmblgrr" (with experience). ?But the help's less-than-
> informative signature tells me I need to review carefully, whereas in
> your approach I would tend to leave it up to the compiler to some
> extent.

I ma not sure to understand the problem. In my approach, you get:

>>>help test
<function <deco>test(*args, **keys)>

then you say, "oh, it's decorated', let's see more..."

>>> help test.func                     # that's the undecorated function
<function test(first=0, last=0)>

Not only, you've got the clear information that the function is
decorate (by the '<deco>' prefix) but also you get a standard and
immediate solution to get the right signature: help(test.func)

I do not understand why this is worse that the <function test(*args,
**keys)> answer provided by the '@wraps' function ?

CS


From stephen at xemacs.org  Wed Apr 27 06:25:40 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 27 Apr 2011 13:25:40 +0900
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTi=ULc_4Hfb97cKm7Ae_HNFiWYLnXw@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<ip7cv9$tck$1@dough.gmane.org>
	<BANLkTimw85KyORHXwSJHuuFH-_j_c=R1TQ@mail.gmail.com>
	<BANLkTim_z7SMV_zf_tvATE9QjbhjmRZjBw@mail.gmail.com>
	<87aafc9zw3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<BANLkTi=ULc_4Hfb97cKm7Ae_HNFiWYLnXw@mail.gmail.com>
Message-ID: <874o5k9v7f.fsf@uwakimon.sk.tsukuba.ac.jp>

Christophe Schlick writes:

 > I do not understand why this is worse that the <function test(*args,
 > **keys)> answer provided by the '@wraps' function ?

It's not.  I misunderstood your proposal.



From ncoghlan at gmail.com  Wed Apr 27 06:26:07 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Apr 2011 14:26:07 +1000
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTi=ULc_4Hfb97cKm7Ae_HNFiWYLnXw@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<ip7cv9$tck$1@dough.gmane.org>
	<BANLkTimw85KyORHXwSJHuuFH-_j_c=R1TQ@mail.gmail.com>
	<BANLkTim_z7SMV_zf_tvATE9QjbhjmRZjBw@mail.gmail.com>
	<87aafc9zw3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<BANLkTi=ULc_4Hfb97cKm7Ae_HNFiWYLnXw@mail.gmail.com>
Message-ID: <BANLkTinOO2Bznt9qZo1iZ12O7LCoCh-g3A@mail.gmail.com>

Attacking (some aspects of) the same problem from a different angle,
note that part of the motivation of PEP 362 (function signature
objects) is to allow functools.wraps to be more effective at
preserving signature information, and to allow similar gains for
functools.partial. It also allows signature checks to be performed
independently of actually calling functions (via the Signature.bind()
method).

As of 3.2, functools.wraps has already been updated to automatically
include a __wrapped__ attribute on the resulting callable, which links
back to the underlying function. The "func" attribute on
functools.partial serves the same purpose.

Actually *writing* decorators is still somewhat cumbersome by default,
but if anything were to happen on that front, the most likely would be
to ask Michele about making the decorators module part of the standard
library.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From cschlick at gmail.com  Wed Apr 27 09:15:54 2011
From: cschlick at gmail.com (Christophe Schlick)
Date: Wed, 27 Apr 2011 09:15:54 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTinOO2Bznt9qZo1iZ12O7LCoCh-g3A@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<ip7cv9$tck$1@dough.gmane.org>
	<BANLkTimw85KyORHXwSJHuuFH-_j_c=R1TQ@mail.gmail.com>
	<BANLkTim_z7SMV_zf_tvATE9QjbhjmRZjBw@mail.gmail.com>
	<87aafc9zw3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<BANLkTi=ULc_4Hfb97cKm7Ae_HNFiWYLnXw@mail.gmail.com>
	<BANLkTinOO2Bznt9qZo1iZ12O7LCoCh-g3A@mail.gmail.com>
Message-ID: <BANLkTi=bu6xn+A9+kpsLHWQcd79SGetabw@mail.gmail.com>

On Wed, Apr 27, 2011 at 6:26 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Attacking (some aspects of) the same problem from a different angle,
> note that part of the motivation of PEP 362 (function signature
> objects) is to allow functools.wraps to be more effective at
> preserving signature information, and to allow similar gains for
> functools.partial. It also allows signature checks to be performed
> independently of actually calling functions (via the Signature.bind()
> method).

Exactly, there are really two complementary solutions to tackle the problem:

1 - copy *all* information from the undecorated function to the
undecorated one. As you say, this requires to extend the '@wraps' tool
to include signature copying (and with function signature objects of
PEP 362 this would be trivial)

2 - let the decorated function as is without copying anything, but
provide a simple, systematic solution to get the
name/signature/docstring of the undecorated function. That's the idea
of my proposal

> Actually *writing* decorators is still somewhat cumbersome by default,
> but if anything were to happen on that front, the most likely would be
> to ask Michele about making the decorators module part of the standard
> library.

As I said in my initial post, there was quite a long discussion in
python-dev exactly about that idea. I finally get the reference to the
topic. Here is the first post of that thread:

http://mail.python.org/pipermail/python-dev/2009-April/088387.html

and here is an extract of what Guido thinks about it:
---
On Wed, Apr 8, 2009 at 7:51 PM, Guido van Rossum <guido at python.org> wrote:

> I also happen to disagree in many cases with decorators that attempt
> to change the signature of the wrapper function to that of the wrapped
> function. While this may make certain kinds of introspection possible,
> again it obscures what's going on to a future maintainer of the code,
> and the cleverness can get in the way of good old-fashioned debugging.

> To me,introspection is mostly useful for certain
> situations like debugging or interactively finding help, but I would
> hesitate to build a large amount of stuff (whether a library,
> framework or application) on systematic use of introspection. In fact,
> I rarely use the inspect module and had to type help(inspect) to
> figure out what you meant by "signature". :-) I guess one reason is
> that in my mind, and in the way I tend to write code, I don't write
> APIs that require introspection -- for example, I don't like APIs that
> do different things when given a "callable" as opposed to something
> else (common practices in web frameworks notwithstanding), and
> thinking about it I would like it even less if an API cared about the
> *actual* signature of a function I pass into it.

Reading that thread was actually the starting point, several months
ago, when I came out with the idea of NSD. The initial question was:
"so, if copying signature is not a step in the right direction, is
there any alternative that offers similar functionality?". In fact,
reducing the boilerplate of decorator pattern only came as a
side-effect

CS


From cschlick at gmail.com  Wed Apr 27 09:40:19 2011
From: cschlick at gmail.com (Christophe Schlick)
Date: Wed, 27 Apr 2011 09:40:19 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTi=bu6xn+A9+kpsLHWQcd79SGetabw@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<ip7cv9$tck$1@dough.gmane.org>
	<BANLkTimw85KyORHXwSJHuuFH-_j_c=R1TQ@mail.gmail.com>
	<BANLkTim_z7SMV_zf_tvATE9QjbhjmRZjBw@mail.gmail.com>
	<87aafc9zw3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<BANLkTi=ULc_4Hfb97cKm7Ae_HNFiWYLnXw@mail.gmail.com>
	<BANLkTinOO2Bznt9qZo1iZ12O7LCoCh-g3A@mail.gmail.com>
	<BANLkTi=bu6xn+A9+kpsLHWQcd79SGetabw@mail.gmail.com>
Message-ID: <BANLkTinmb0e+O21c7nijV4fqnbgJ-gkx8A@mail.gmail.com>

Another snippet of the BDFL going in the same direction:

On Fri, Apr 10, 2009 at 7:55 PM, Guido van Rossum <guido at python.org> wrote:

> But seeing the decorator is often essential for understanding what
> goes on! Even if the decorator preserves the signature (in truth or
> according inspect), many decorators *do* something, and it's important
> to know how a function is decorated. For example, I work a lot with a
> small internal framework at Google whose decorators can raise
> exceptions and set instance variables; they also help me understand
> under which conditions a method can be called.

That's one of the reason for which I've added the recursive
'self.deco' reference in the decorated function, to get the complete
info about all chained decorators applied on a given function. Here
again, I am not aware of any implementation that offers a similar
feature.

CS


From grosser.meister.morti at gmx.net  Thu Apr 28 07:18:35 2011
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Thu, 28 Apr 2011 07:18:35 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTinLSHnS4ef6AHcLs2S_XnGfgfT_UA@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>	<BANLkTikD8mYM+8DG-FyHt1usgD1_hseV=g@mail.gmail.com>
	<BANLkTinLSHnS4ef6AHcLs2S_XnGfgfT_UA@mail.gmail.com>
Message-ID: <4DB8F8AB.6080807@gmx.net>

Just before I go to bed, here some alternative implementations:

Usage:
------
# for argument-less decorators:
@decorator
def my_deco(func,*func_args,**func_kwargs):
	pass

# of course regular arguments can also be declared (same for all funcs below)
@my_deco
def func(*func_args,**func_kwargs):
	pass

# for decorators with arguments:
# even when there are no default arguments function-call parenthesis are needed
@decorator_with_args('foo',bar='baz')
def my_deco(func,deco_args,deco_kwargs,*func_args,**func_kwargs):
	pass

@my_deco(*deco_args,**deco_kwargs)
def func(*func_args,**func_kwargs):
	pass

# alternative version where the decorator arguments are expanded:
# `...` is a placeholder for the arguments of the decorator in regular argument syntax.
# This way the decorator arguments can be declared inline and no deco_(kw)args or self.*
# is needed. Also decorator arguments are not decoupled from their default values this way.
@decorator_with_expanded_args
def my_deco(func,...,*func_args,**func_kwargs):
	pass

@my_deco(*deco_args,**deco_kwargs)
def func(*func_args,**func_kwargs):
	pass


Implementation:
---------------
from types import FunctionType, ClassType
from functools import wraps

def decorator(deco):
	@wraps(deco)
	def _deco(func):
		@wraps(func)
		def _f(*args,**kwargs):
			return deco(func,*args,**kwargs)
		return _f
	return _deco

def decorator_with_args(*deco_default_args,**deco_default_kwargs):
	def _deco_deco_deco(deco):
		@wraps(deco)
		def _deco_deco(*deco_args,**deco_kwargs):
			if len(deco_args) < len(deco_default_args):
				deco_args = deco_args+deco_default_args[len(deco_args):]
			merged_deco_kwargs = dict(deco_default_kwargs)
			merged_deco_kwargs.update(deco_kwargs)
			del deco_kwargs
			def _deco(func):
				@wraps(func)
				def _f(*args,**kwargs):
					return deco(
						func,deco_args,merged_deco_kwargs,*args,**kwargs)
				return _f
			return _deco
		return _deco_deco
	return _deco_deco_deco

def decorator_with_expanded_args(deco):
	if isinstance(deco, FunctionType):
		co = deco.func_code
		deco_name = deco.func_name
		arg_names = list(co.co_varnames[0:co.co_argcount])
	elif isinstance(deco, ClassType):
		co = deco.__init__.func_code
		deco_name = deco.__name__
		arg_names = list(co.co_varnames[1:co.co_argcount])
	elif hasattr(deco, '__call__'):
		co = deco.__call__.func_code
		deco_name = type(deco).__name__
		arg_names = list(co.co_varnames[0:co.co_argcount])
	else:
		raise TypeError('not a decorateable object')
	if not arg_names:
		raise TypeError('decorator function needs a func argument')
	del co
	del arg_names[0]
	min_argcount = len(arg_names)
	if deco.func_defaults:
		min_argcount -= len(deco.func_defaults)
	@wraps(deco)
	def _deco_deco(*args,**kwargs):
		deco_args = list(args)
		n = len(deco_args)
		if n < len(arg_names):
			i = n - min_argcount
			for arg in arg_names[n:]:
				if arg in kwargs:
					deco_args.append(kwargs.pop(arg))
				elif i < 0:
					raise TypeError(
						'%s() takes at least %d positional ' +
						'arguments (%d given)' %
						(deco_name, min_argcount, len(deco_args)))
				else:
					deco_args.append(deco.func_defaults[i])
				i += 1
		if kwargs:
			arg = kwargs.keys()[0]
			if arg in arg_names:
				raise TypeError(
					"%s() got multiple values for keyword argument '%s'" %
					(deco_name, arg))
			else:
				raise TypeError("%s() got an unexpected keyword argument '%s'" %
					(deco_name, arg))
		deco_args = tuple(deco_args)
		def _deco(func):
			@wraps(func)
			def _f(*args,**kwargs):
				return deco(func,*(deco_args+args),**kwargs)
			return _f
		return _deco
	return _deco_deco


What do you think?

	-panzi


From szport at gmail.com  Thu Apr 28 09:24:24 2011
From: szport at gmail.com (ZS)
Date: Thu, 28 Apr 2011 11:24:24 +0400
Subject: [Python-ideas] Equality of same NaN instances?
Message-ID: <BANLkTik+4-7d1jQhBQep8F_1ouAOd7wV-Q@mail.gmail.com>

Last debate on python-dev about behavior NaNs in containers raised
another important question:

Should the instance of NaN be equal to itself?

P.S. Of course I suppose that different instances of NaN do not have
to be equal.

---
Zaur


From cmjohnson.mailinglist at gmail.com  Thu Apr 28 10:08:06 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Wed, 27 Apr 2011 22:08:06 -1000
Subject: [Python-ideas] Equality of same NaN instances?
In-Reply-To: <BANLkTik+4-7d1jQhBQep8F_1ouAOd7wV-Q@mail.gmail.com>
References: <BANLkTik+4-7d1jQhBQep8F_1ouAOd7wV-Q@mail.gmail.com>
Message-ID: <BANLkTikqiJSEK1DSC+aCGZdkfrqn76qGdA@mail.gmail.com>

This has been discussed before. See
http://mail.python.org/pipermail/python-ideas/2010-March/thread.html#6945
etc. for background on the issue.

My two cents? Follow what C does/the IEEE spec unless there's a really
compelling reason to do something else.


From alexander.belopolsky at gmail.com  Thu Apr 28 10:43:07 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 28 Apr 2011 04:43:07 -0400
Subject: [Python-ideas] Equality of same NaN instances?
In-Reply-To: <BANLkTikqiJSEK1DSC+aCGZdkfrqn76qGdA@mail.gmail.com>
References: <BANLkTik+4-7d1jQhBQep8F_1ouAOd7wV-Q@mail.gmail.com>
	<BANLkTikqiJSEK1DSC+aCGZdkfrqn76qGdA@mail.gmail.com>
Message-ID: <BANLkTi=fWaP=VHqJThsqcpVmyMFthxiqgA@mail.gmail.com>

On Thu, Apr 28, 2011 at 4:08 AM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
..
> My two cents? Follow what C does/the IEEE spec unless there's a really
> compelling reason to do something else.

One good thing about standards is that there are plenty to choose
from.  IMO, Python's "float" type is much closer to Java's "Double"
than to machine FP types standardized by IEEE 754.

If nothing else, annual reoccurrence of long threads on this topic is
a reason enough to reconsider which standard to follow.


From alexander.belopolsky at gmail.com  Thu Apr 28 10:52:34 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 28 Apr 2011 04:52:34 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
Message-ID: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>

Another spin-off from the "[Python-Dev] PyObject_RichCompareBool
identity shortcut" thread:

> I would like to discuss another peculiarity of NaNs:
>
>>>> float('nan') < 0
> False
>>>> float('nan') > 0
> False
>
> This property in my experience causes much more trouble than nan ==
> nan being false. ?The problem is that common sorting or binary search
> algorithms may degenerate into infinite loops in the presence of nans.
> ?This may even happen when searching for a finite value in a large
> array that contains a single nan. ?Errors like this do happen in the
> wild and and after chasing a bug like this programmers tend to avoid
> nans at all costs. ?Oftentimes this leads to using "magic"
> placeholders such as 1e300 for missing data.
>
> Since py3k has already made None < 0 an error, it may be reasonable
> for float('nan') < 0 to raise an error as well (probably ValueError
> rather than TypeError). ?This will not make lists with nans sortable
> or searchable using binary search, but will make associated bugs
> easier to find.
>


From ziade.tarek at gmail.com  Thu Apr 28 14:27:14 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 28 Apr 2011 14:27:14 +0200
Subject: [Python-ideas] [Python-Dev] the role of assert in the standard
	library ?
In-Reply-To: <4DB94102.9020701@voidspace.org.uk>
References: <BANLkTimtoxmyNg9sn7e27soPBdxQMZ-q2Q@mail.gmail.com>
	<ipb8qr$6jl$1@dough.gmane.org> <4DB94102.9020701@voidspace.org.uk>
Message-ID: <BANLkTi=Ba-T8RHJ0e4_967qoc=65KsBXdw@mail.gmail.com>

This is a thread from python-dev I am moving to python-ideas, because
I want to debate the "assert" keyword :)

On Thu, Apr 28, 2011 at 12:27 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> On 28/04/2011 09:34, Terry Reedy wrote:
>>
>> On 4/28/2011 3:54 AM, Tarek Ziad? wrote:
>>>
>>> Hello
>>>
>>> I removed some assert calls in distutils some time ago because the
>>> package was not behaving correctly when people were using Python with
>>> the --optimize flag. In other words, assert became a full part of the
>>> code logic and removing them via -O was changing the behavior.
>>>
>>> In my opinion assert should be avoided completely anywhere else than
>>> in the tests. If this is a wrong statement, please let me know why :)
>>
>> My understanding is that assert can be used in production code but only to
>> catch logic errors by testing supposed invariants or postconditions. It
>> should not be used to test usage errors, including preconditions. In other
>> words, assert presence or absence should not affect behavior unless the code
>> has a bug.

But it does affect the behaviour at the end: when the code has a bug,
then the way the code works is affected and differs depending if -O is
used.

Let me take an example:

bucket = []

def add(stuff):
    bucket.append(stuff)

def purge_and_do_something():
    bucket.clear()
    assert len(bucket) == 0
    ... do something by being sure the bucket is empty ...


here, we could say assert is legitimate, as it just checks a
post-condition. But this code is not thread-safe and if it's run via
several threads, some code could add something in the bucket while
purge() check for the assertion.

So, if I run this code using -O it will not behave the same: it will
seem to work. I am not arguing against the fact that this code should
be changed and set up a lock.

My point is that I do not understand why there are assert calls to
check post-conditions. Those seem to be relics from the developer that
marked something she needed to take care of in her code, but did not
yet. e.g. like a TODO or a XXX or a NotImplementedError

Moreover, why -O and -OO are removing assertions in that case ? even
if the assert is "supposed not to happen" it's present in the code and
can happen (or well, remove it). So "optimize" potentially changes the
behaviour of the code.

>From what I understood so far, a line with an assert should be
considered as a dead code if we want the code to behave always the
same way. I remove dead code in my code...


Cheers
Tarek


-- 
Tarek Ziad? | http://ziade.org


From mal at egenix.com  Thu Apr 28 14:49:22 2011
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 28 Apr 2011 14:49:22 +0200
Subject: [Python-ideas] [Python-Dev] the role of assert in the standard
 library ?
In-Reply-To: <BANLkTi=Ba-T8RHJ0e4_967qoc=65KsBXdw@mail.gmail.com>
References: <BANLkTimtoxmyNg9sn7e27soPBdxQMZ-q2Q@mail.gmail.com>	<ipb8qr$6jl$1@dough.gmane.org>
	<4DB94102.9020701@voidspace.org.uk>
	<BANLkTi=Ba-T8RHJ0e4_967qoc=65KsBXdw@mail.gmail.com>
Message-ID: <4DB96252.6050002@egenix.com>

Tarek Ziad? wrote:
> This is a thread from python-dev I am moving to python-ideas, because
> I want to debate the "assert" keyword :)
> 
> On Thu, Apr 28, 2011 at 12:27 PM, Michael Foord
> <fuzzyman at voidspace.org.uk> wrote:
>> On 28/04/2011 09:34, Terry Reedy wrote:
>>>
>>> On 4/28/2011 3:54 AM, Tarek Ziad? wrote:
>>>>
>>>> Hello
>>>>
>>>> I removed some assert calls in distutils some time ago because the
>>>> package was not behaving correctly when people were using Python with
>>>> the --optimize flag. In other words, assert became a full part of the
>>>> code logic and removing them via -O was changing the behavior.
>>>>
>>>> In my opinion assert should be avoided completely anywhere else than
>>>> in the tests. If this is a wrong statement, please let me know why :)
>>>
>>> My understanding is that assert can be used in production code but only to
>>> catch logic errors by testing supposed invariants or postconditions. It
>>> should not be used to test usage errors, including preconditions. In other
>>> words, assert presence or absence should not affect behavior unless the code
>>> has a bug.
> 
> But it does affect the behaviour at the end: when the code has a bug,
> then the way the code works is affected and differs depending if -O is
> used.

"assert"s are meant to easily and transparently include testing
code in production code, without affecting the production
version's performance when run with the -O flag.

In your example, you can assume that your code does
indeed do what it's meant to do in production code, because
you will have tested the code in your test environment.

Assuming that you don't allow untested code to run on a production
system, you can then safely remove the assert bytecodes from
the program and avoid the added overhead using the -O flag.

For longer pieces of testing code, you can use:

if __debug__:
    print("Testing ...")

This code will also get removed by the -O flag.

Both methods allow testing complex code without having to
duplicate much of the code in test cases, just to check
certain corner cases.

> Let me take an example:
> 
> bucket = []
> 
> def add(stuff):
>     bucket.append(stuff)
> 
> def purge_and_do_something():
>     bucket.clear()
>     assert len(bucket) == 0
>     ... do something by being sure the bucket is empty ...
> 
> 
> here, we could say assert is legitimate, as it just checks a
> post-condition. But this code is not thread-safe and if it's run via
> several threads, some code could add something in the bucket while
> purge() check for the assertion.
> 
> So, if I run this code using -O it will not behave the same: it will
> seem to work. I am not arguing against the fact that this code should
> be changed and set up a lock.
> 
> My point is that I do not understand why there are assert calls to
> check post-conditions. Those seem to be relics from the developer that
> marked something she needed to take care of in her code, but did not
> yet. e.g. like a TODO or a XXX or a NotImplementedError
> 
> Moreover, why -O and -OO are removing assertions in that case ? even
> if the assert is "supposed not to happen" it's present in the code and
> can happen (or well, remove it). So "optimize" potentially changes the
> behaviour of the code.
> 
>>From what I understood so far, a line with an assert should be
> considered as a dead code if we want the code to behave always the
> same way. I remove dead code in my code...

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 28 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2011-06-20: EuroPython 2011, Florence, Italy               53 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/


From steve at pearwood.info  Thu Apr 28 15:49:40 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 28 Apr 2011 23:49:40 +1000
Subject: [Python-ideas] [Python-Dev] the role of assert in the standard
 library ?
In-Reply-To: <BANLkTi=Ba-T8RHJ0e4_967qoc=65KsBXdw@mail.gmail.com>
References: <BANLkTimtoxmyNg9sn7e27soPBdxQMZ-q2Q@mail.gmail.com>	<ipb8qr$6jl$1@dough.gmane.org>
	<4DB94102.9020701@voidspace.org.uk>
	<BANLkTi=Ba-T8RHJ0e4_967qoc=65KsBXdw@mail.gmail.com>
Message-ID: <4DB97074.3040105@pearwood.info>

Tarek Ziad? wrote:

> From what I understood so far, a line with an assert should be
> considered as a dead code if we want the code to behave always the
> same way. I remove dead code in my code...

No. There's a difference between dead code (code that cannot be reached) 
and asserts. An assert is a statement of confidence:

"I'm sure that this assertion is true, but I'm not 100% sure, because 
bugs do exist. But since I'm *nearly* 100% sure, it is safe to optimize 
the assertions away, *if* you are brave enough to run with the -O flag."

If you've ever written the typical defensive pattern:

if a:
    do_this()
elif b:
    do_that()
elif c:
    do_something_else()
else:
    # This can never happen.
    raise RuntimeError('unexpected error')

then you have a perfect candidate for an assertion:

assert any([a, b, c], 'unexpected error')


You know what they say about things that can never happen: they *do* 
happen, more often than you like. An assertion is to check for things 
that can never happen. Since it can't happen, it's safe to optimize it 
away and not perform the check. But since things that can't happen do 
happen (due to logic errors and the presence of unexpected bugs), it's 
better to generate a failure immediately, where you perform the check, 
rather than somewhere else far distant from the problem, or worse, 
returning the wrong result.

Assertions are a case of practicality beats purity: in a perfect world, 
we'd always be 100% confident that our code was bug-free and free of any 
logic errors, and that our assumptions were perfectly correct. But in 
the real world, we can't always be quite so sure, and asserts cover that 
middle ground where we're *almost* sure about a condition, but not 
entirely. Or perhaps we're merely paranoid. Either way, asserts should 
never be something you *rely* on (e.g. checking user input).

Let me give you a real world example: I have some code to calculate the 
Pearson's Correlation Coefficient, r. At the end of the function, just 
before returning, I assert -1 <= r <= 1. I've run this function 
thousands of times, possibly tens of thousands, over periods of months, 
without any problems. Last week I got an assertion error: r was 
something like 1.00000000000000001 (or thereabouts).

So now I know there's a bug in my code. (Unfortunately, not *where* it 
is.) Without the assert, I wouldn't know, because that r would likely 
have been used in some larger calculation, without me ever noticing that 
it can be out of range.




-- 
Steven



From steve at pearwood.info  Thu Apr 28 16:01:53 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 29 Apr 2011 00:01:53 +1000
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
Message-ID: <4DB97351.2050605@pearwood.info>

Alexander Belopolsky wrote:
> Another spin-off from the "[Python-Dev] PyObject_RichCompareBool
> identity shortcut" thread:
> 
>> I would like to discuss another peculiarity of NaNs:
>>
>>>>> float('nan') < 0
>> False
>>>>> float('nan') > 0
>> False
>>
>> This property in my experience causes much more trouble than nan ==
>> nan being false.  The problem is that common sorting or binary search
>> algorithms may degenerate into infinite loops in the presence of nans.

I think I would like to see a demonstration of this rather than just 
take your word for it.

 >>> sorted([4, 5, 1, float('nan'), 3, 2])
[1, 2, 3, 4, 5, nan]




-- 
Steven



From mikegraham at gmail.com  Thu Apr 28 16:22:40 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Thu, 28 Apr 2011 10:22:40 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
Message-ID: <BANLkTiniGWnPcM3v+e6_N=0QhsxoK+ejow@mail.gmail.com>

On Thu, Apr 28, 2011 at 4:52 AM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> Another spin-off from the "[Python-Dev] PyObject_RichCompareBool
> identity shortcut" thread:
>
>> I would like to discuss another peculiarity of NaNs:
>>
>>>>> float('nan') < 0
>> False
>>>>> float('nan') > 0
>> False
>>
>> This property in my experience causes much more trouble than nan ==
>> nan being false. ?The problem is that common sorting or binary search
>> algorithms may degenerate into infinite loops in the presence of nans.
>> ?This may even happen when searching for a finite value in a large
>> array that contains a single nan. ?Errors like this do happen in the
>> wild and and after chasing a bug like this programmers tend to avoid
>> nans at all costs. ?Oftentimes this leads to using "magic"
>> placeholders such as 1e300 for missing data.
>>
>> Since py3k has already made None < 0 an error, it may be reasonable
>> for float('nan') < 0 to raise an error as well (probably ValueError
>> rather than TypeError). ?This will not make lists with nans sortable
>> or searchable using binary search, but will make associated bugs
>> easier to find.

I'm -0 on this -- I really favor having NaNs behave like NaNs.
Obviously this is a weird fit for Python, but so what? Python does its
best never to give you NaNs. If you've done something to get a NaN
it's because of a library bug or because you really wanted the NaNs
and should know what you're doing.


From mikegraham at gmail.com  Thu Apr 28 16:22:54 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Thu, 28 Apr 2011 10:22:54 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <4DB97351.2050605@pearwood.info>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
Message-ID: <BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>

On Thu, Apr 28, 2011 at 10:01 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> I think I would like to see a demonstration of this rather than just take
> your word for it.

One demonstration would be

def bubble_sort(xs):
    while True:
        changed = False
        for i in range(len(xs) - 1):
            if not (xs[i] < xs[i + 1]):
                changed = True
                xs[i], xs[i + 1] = xs[i + 1], xs[i]
        if not changed:
            break

bubble_sort([float('nan)'] * 2)


From alexander.belopolsky at gmail.com  Thu Apr 28 16:42:20 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 28 Apr 2011 10:42:20 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <4DB97351.2050605@pearwood.info>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
Message-ID: <BANLkTim3g6xuh4aMtYQ00CtUGhzO3G5O6Q@mail.gmail.com>

On Thu, Apr 28, 2011 at 10:01 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> Alexander Belopolsky wrote:
..
> I think I would like to see a demonstration of this rather than just take
> your word for it.
>
>>>> sorted([4, 5, 1, float('nan'), 3, 2])
> [1, 2, 3, 4, 5, nan]

Hmm, a quick search of the tracker yielded issue7915 which
demonstrates how presence of nans causes sort to leave a list
unsorted.   Using binary search on unsorted data leads to nonsensical
results, but I don't seem to be able to produce infinite loops with
python's bisect.  Maybe I saw nan-caused infinite loops  in some other
libraries.  I learned long ago to rid my data of NaNs before doing any
type of comparison, so my recollection of the associated problems is
admittedly vague.  I'll try to come up with something, though.

http://bugs.python.org/issue7915


From alexander.belopolsky at gmail.com  Thu Apr 28 17:02:09 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 28 Apr 2011 11:02:09 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTiniGWnPcM3v+e6_N=0QhsxoK+ejow@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<BANLkTiniGWnPcM3v+e6_N=0QhsxoK+ejow@mail.gmail.com>
Message-ID: <BANLkTimsz=GnUvPxfMhyqCL0OMrXe3b8eA@mail.gmail.com>

On Thu, Apr 28, 2011 at 10:22 AM, Mike Graham <mikegraham at gmail.com> wrote:
> On Thu, Apr 28, 2011 at 4:52 AM, Alexander Belopolsky
> <alexander.belopolsky at gmail.com> wrote:
..
>>> Since py3k has already made None < 0 an error, it may be reasonable
>>> for float('nan') < 0 to raise an error as well (probably ValueError
>>> rather than TypeError). ?This will not make lists with nans sortable
>>> or searchable using binary search, but will make associated bugs
>>> easier to find.
>
> I'm -0 on this -- I really favor having NaNs behave like NaNs.

.. but IEEE 754 specifies that NaNs compare as "unordered".


From guido at python.org  Thu Apr 28 17:44:08 2011
From: guido at python.org (Guido van Rossum)
Date: Thu, 28 Apr 2011 08:44:08 -0700
Subject: [Python-ideas] Equality of same NaN instances?
In-Reply-To: <BANLkTikqiJSEK1DSC+aCGZdkfrqn76qGdA@mail.gmail.com>
References: <BANLkTik+4-7d1jQhBQep8F_1ouAOd7wV-Q@mail.gmail.com>
	<BANLkTikqiJSEK1DSC+aCGZdkfrqn76qGdA@mail.gmail.com>
Message-ID: <BANLkTikABaYmTzGDN3Mn0=Pdc9agJ+V6xA@mail.gmail.com>

On Thu, Apr 28, 2011 at 1:08 AM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
> This has been discussed before. See
> http://mail.python.org/pipermail/python-ideas/2010-March/thread.html#6945
> etc. for background on the issue.
>
> My two cents? Follow what C does/the IEEE spec unless there's a really
> compelling reason to do something else.

Ah, the compelling difference is that the IEEE spec only talks about
values, whereas Python also has to have an opinion about object.

-- 
--Guido van Rossum (python.org/~guido)


From python at mrabarnett.plus.com  Thu Apr 28 18:00:29 2011
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 28 Apr 2011 17:00:29 +0100
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTimsz=GnUvPxfMhyqCL0OMrXe3b8eA@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>	<BANLkTiniGWnPcM3v+e6_N=0QhsxoK+ejow@mail.gmail.com>
	<BANLkTimsz=GnUvPxfMhyqCL0OMrXe3b8eA@mail.gmail.com>
Message-ID: <4DB98F1D.4070601@mrabarnett.plus.com>

On 28/04/2011 16:02, Alexander Belopolsky wrote:
> On Thu, Apr 28, 2011 at 10:22 AM, Mike Graham<mikegraham at gmail.com>  wrote:
>> On Thu, Apr 28, 2011 at 4:52 AM, Alexander Belopolsky
>> <alexander.belopolsky at gmail.com>  wrote:
> ..
>>>> Since py3k has already made None<  0 an error, it may be reasonable
>>>> for float('nan')<  0 to raise an error as well (probably ValueError
>>>> rather than TypeError).  This will not make lists with nans sortable
>>>> or searchable using binary search, but will make associated bugs
>>>> easier to find.
>>
>> I'm -0 on this -- I really favor having NaNs behave like NaNs.
>
> .. but IEEE 754 specifies that NaNs compare as "unordered".
>
I get this (Python 3.1.2):

 >>> sorted([1, float('nan'), 0, 0])
[1, nan, 0, 0]
 >>> sorted([1, 0, float('nan'), 0])
[0, 0, 1, nan]

I think that as NaNs behave like this:

 >>> float('nan') == float('nan')
False
 >>> float('nan') < float('nan')
False
 >>> float('nan') > float('nan')
False

trying to sort them should raise an exception, just to preserve users'
sanity!

IMHO, the current behaviour just makes it look like a bug.


From steve at pearwood.info  Thu Apr 28 18:00:27 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 29 Apr 2011 02:00:27 +1000
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
Message-ID: <4DB98F1B.8060700@pearwood.info>

Mike Graham wrote:
> On Thu, Apr 28, 2011 at 10:01 AM, Steven D'Aprano <steve at pearwood.info> wrote:
>> I think I would like to see a demonstration of this rather than just take
>> your word for it.
> 
> One demonstration would be
[snip]

Thank you.

Nevertheless, that does appear to be an easy fix:


def bubble_sort(xs):
     while True:
         changed = False
         for i in range(len(xs) - 1):
             # don't use `not (xs[i] < xs[i + 1])` as that fails in the
             # presence of NANs
             if xs[i] >= xs[i + 1]:
                 changed = True
                 xs[i], xs[i + 1] = xs[i + 1], xs[i]
         if not changed:
             break



-- 
Steven



From alexander.belopolsky at gmail.com  Thu Apr 28 18:12:21 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 28 Apr 2011 12:12:21 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
Message-ID: <BANLkTi=qd=UNcsrVL6cnx8pS4BXmUdKPFg@mail.gmail.com>

On Thu, Apr 28, 2011 at 4:52 AM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
..
>> Since py3k has already made None < 0 an error, it may be reasonable
>> for float('nan') < 0 to raise an error as well (probably ValueError
>> rather than TypeError). ?This will not make lists with nans sortable
>> or searchable using binary search, but will make associated bugs
>> easier to find.
>>
>

Furthermore, IEEE 754 specifies exactly what I propose:

"""
IEEE 754 assigns values to all relational expressions involving NaN.
In the syntax of C , the predicate x != y is True but all others, x <
y , x <= y , x == y , x >= y and x > y, are False whenever x or y or
both are NaN, and then all but x != y and x == y are INVALID
operations too and must so signal.
"""
-- Lecture Notes on the Status of IEEE Standard 754 for Binary
Floating-Point Arithmetic by Prof. W. Kahan
http://www.cs.berkeley.edu/~wkahan/ieee754status/ieee754.ps

The problem with faithfully implementing IEEE 754 in Python is that
exceptions in IEEE standard don't have the same meaning as in Python.
 IEEE 754 requires that a value is computed even when the operation
signals an exception.  The program can then decide whether to
terminate computation or propagate the value. In Python, we have to
choose between raising an exception and returning the value.  We
cannot have both.  It appears that in most cases IEEE 754 "INVALID"
exception is treated as a terminating exception by Python and
operations that signal INVALID in IEEE 754 raise an exception in
Python.  Therefore making <, >, etc. raise on NaN while keeping the
status quo for != and == would bring Python floats closer to
compliance with IEEE 754.


From rob.cliffe at btinternet.com  Thu Apr 28 18:17:46 2011
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Thu, 28 Apr 2011 17:17:46 +0100
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <4DB98F1B.8060700@pearwood.info>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>	<4DB97351.2050605@pearwood.info>	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
Message-ID: <4DB9932A.9070903@btinternet.com>



On 28/04/2011 17:00, Steven D'Aprano wrote:
> Mike Graham wrote:
>> On Thu, Apr 28, 2011 at 10:01 AM, Steven D'Aprano 
>> <steve at pearwood.info> wrote:
>>> I think I would like to see a demonstration of this rather than just 
>>> take
>>> your word for it.
>>
>> One demonstration would be
> [snip]
>
> Thank you.
>
> Nevertheless, that does appear to be an easy fix:
>
>
> def bubble_sort(xs):
>     while True:
>         changed = False
>         for i in range(len(xs) - 1):
>             # don't use `not (xs[i] < xs[i + 1])` as that fails in the
>             # presence of NANs
>             if xs[i] >= xs[i + 1]:
>                 changed = True
>                 xs[i], xs[i + 1] = xs[i + 1], xs[i]
>         if not changed:
>             break
>
>
>
True.  But why be forced to walk on eggshells when writing a perfectly 
ordinary bit of code that "ought" to work as is?
Rob Cliffe


From robert.kern at gmail.com  Thu Apr 28 18:46:10 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 28 Apr 2011 11:46:10 -0500
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTimsz=GnUvPxfMhyqCL0OMrXe3b8eA@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>	<BANLkTiniGWnPcM3v+e6_N=0QhsxoK+ejow@mail.gmail.com>
	<BANLkTimsz=GnUvPxfMhyqCL0OMrXe3b8eA@mail.gmail.com>
Message-ID: <ipc5kj$m41$1@dough.gmane.org>

On 4/28/11 10:02 AM, Alexander Belopolsky wrote:
> On Thu, Apr 28, 2011 at 10:22 AM, Mike Graham<mikegraham at gmail.com>  wrote:
>> On Thu, Apr 28, 2011 at 4:52 AM, Alexander Belopolsky
>> <alexander.belopolsky at gmail.com>  wrote:
> ..
>>>> Since py3k has already made None<  0 an error, it may be reasonable
>>>> for float('nan')<  0 to raise an error as well (probably ValueError
>>>> rather than TypeError).  This will not make lists with nans sortable
>>>> or searchable using binary search, but will make associated bugs
>>>> easier to find.
>>
>> I'm -0 on this -- I really favor having NaNs behave like NaNs.
>
> .. but IEEE 754 specifies that NaNs compare as "unordered".

Not quite, IIRC. I don't have it in front of me, but I do recall that it 
specifies how it behaves in two different situations:

1. Where you have a comparison function that returns the relationship between 
the two operands, IEEE-754 specifies that in addition to GT, LT, and EQ, you 
ought to include "unordered" to use when a NaN is involved.

2. Where you have comparison operators like <, ==, etc. that return bools, NaNs 
will return False for all comparisons.

They may specify whether or not FPE signals should be issued, I don't recall, 
but I suspect that if they are quiet NaNs, they won't issue a SIGFPE. 
Higher-level exceptions were not contemplated by IEEE-754, IIRC.

Python uses the < operator for sorting, not a comparison function, so it's 
current behavior is perfectly in line with the IEEE-754 spec.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From alexander.belopolsky at gmail.com  Thu Apr 28 19:00:31 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 28 Apr 2011 13:00:31 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <ipc5kj$m41$1@dough.gmane.org>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<BANLkTiniGWnPcM3v+e6_N=0QhsxoK+ejow@mail.gmail.com>
	<BANLkTimsz=GnUvPxfMhyqCL0OMrXe3b8eA@mail.gmail.com>
	<ipc5kj$m41$1@dough.gmane.org>
Message-ID: <BANLkTimMz_wJyiZ_X=40J+Df=C5gNbTrOg@mail.gmail.com>

On Thu, Apr 28, 2011 at 12:46 PM, Robert Kern <robert.kern at gmail.com> wrote:
..
> Python uses the < operator for sorting, not a comparison function, so it's
> current behavior is perfectly in line with the IEEE-754 spec.

No, it is not.  As I explained in the previous post, IEEE-754
prescribes different behavior for <, >, <=, and >= operations and !=
and ==.  The former signal INVALID exception while the later don't.
Python does not make this distinction.


From mikegraham at gmail.com  Thu Apr 28 19:26:38 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Thu, 28 Apr 2011 13:26:38 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <4DB98F1B.8060700@pearwood.info>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
Message-ID: <BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>

On Thu, Apr 28, 2011 at 12:00 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Mike Graham wrote:
>>
>> On Thu, Apr 28, 2011 at 10:01 AM, Steven D'Aprano <steve at pearwood.info>
>> wrote:
>>>
>>> I think I would like to see a demonstration of this rather than just take
>>> your word for it.
>>
>> One demonstration would be
>
> [snip]
>
> Thank you.
>
> Nevertheless, that does appear to be an easy fix:
>
>
> def bubble_sort(xs):
> ? ?while True:
> ? ? ? ?changed = False
> ? ? ? ?for i in range(len(xs) - 1):
> ? ? ? ? ? ?# don't use `not (xs[i] < xs[i + 1])` as that fails in the
> ? ? ? ? ? ?# presence of NANs
> ? ? ? ? ? ?if xs[i] >= xs[i + 1]:
> ? ? ? ? ? ? ? ?changed = True
> ? ? ? ? ? ? ? ?xs[i], xs[i + 1] = xs[i + 1], xs[i]
> ? ? ? ?if not changed:
> ? ? ? ? ? ?break
>
> --
> Steven

Note this actually isn't an improvement--it merely takes a noticeable
error and turns it into a data-polluter. (Sorting a sequence
containing NaNs is obviously not a valid operation, which is the
argument for OP's suggestion.)

MG


From alexander.belopolsky at gmail.com  Thu Apr 28 21:25:42 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 28 Apr 2011 15:25:42 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
Message-ID: <BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>

I posted a patch implementing this proposal on the tracker:

http://bugs.python.org/issue11949

Interestingly, the only substantive change that was needed to pass the
test suit revealed a bug in the test logic.  The tests in
cmath_testcases.txt include testing for -0.0 results, but the
processing in test_math.py ignores the difference between 0.0 and
-0.0.  For example, test_math will still pass if you make the
following change:

--- a/Lib/test/cmath_testcases.txt
+++ b/Lib/test/cmath_testcases.txt
@@ -405,7 +405,7 @@
 -- zeros
 asin0000 asin 0.0 0.0 -> 0.0 0.0
 asin0001 asin 0.0 -0.0 -> 0.0 -0.0
-asin0002 asin -0.0 0.0 -> -0.0 0.0
+asin0002 asin -0.0 0.0 -> 0.0 0.0
 asin0003 asin -0.0 -0.0 -> -0.0 -0.0


From guido at python.org  Fri Apr 29 00:10:24 2011
From: guido at python.org (Guido van Rossum)
Date: Thu, 28 Apr 2011 15:10:24 -0700
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
Message-ID: <BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>

On Thu, Apr 28, 2011 at 12:25 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> I posted a patch implementing this proposal on the tracker:
>
> http://bugs.python.org/issue11949

Interesting indeed! I'd like to hear from the numpy folks about this.

But isn't a similar change needed for Decimal?

-- 
--Guido van Rossum (python.org/~guido)


From robert.kern at gmail.com  Fri Apr 29 00:13:33 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 28 Apr 2011 17:13:33 -0500
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTimMz_wJyiZ_X=40J+Df=C5gNbTrOg@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>	<BANLkTiniGWnPcM3v+e6_N=0QhsxoK+ejow@mail.gmail.com>	<BANLkTimsz=GnUvPxfMhyqCL0OMrXe3b8eA@mail.gmail.com>	<ipc5kj$m41$1@dough.gmane.org>
	<BANLkTimMz_wJyiZ_X=40J+Df=C5gNbTrOg@mail.gmail.com>
Message-ID: <ipcoqd$9oe$1@dough.gmane.org>

On 4/28/11 12:00 PM, Alexander Belopolsky wrote:
> On Thu, Apr 28, 2011 at 12:46 PM, Robert Kern<robert.kern at gmail.com>  wrote:
> ..
>> Python uses the<  operator for sorting, not a comparison function, so it's
>> current behavior is perfectly in line with the IEEE-754 spec.
>
> No, it is not.  As I explained in the previous post, IEEE-754
> prescribes different behavior for<,>,<=, and>= operations and !=
> and ==.  The former signal INVALID exception while the later don't.
> Python does not make this distinction.

But it also states that such signals should *not* trap by default. The only 
thing I can really fault Python for, compliance-wise, is that it will hide the 
FPE from being handled in user code because of the 
PyFPE_START_PROTECT/PyFPE_END_PROTECT macros that surround the actual C 
operation. The only way to get the FPE to handle it is to build and install 
fpectl, which is officially discouraged.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From anikom15 at gmail.com  Fri Apr 29 00:38:43 2011
From: anikom15 at gmail.com (Westley =?iso-8859-1?Q?Mart=EDnez?=)
Date: Thu, 28 Apr 2011 15:38:43 -0700
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <BADC24AD-AF0C-405E-8B9D-CAA739B28828@masklinn.net>
References: <4DB5CD47.3060808@interia.pl>
	<BANLkTimKL6nyP+0Q33BMj209UU-3Z02-=Q@mail.gmail.com>
	<BANLkTike9TLrGW4oxes5-O3qCpcAwXaW9A@mail.gmail.com>
	<BANLkTininhh5B8WCubc_3TRCJLx7=zRjQA@mail.gmail.com>
	<BADC24AD-AF0C-405E-8B9D-CAA739B28828@masklinn.net>
Message-ID: <20110428223842.GB2663@Smoke>

On Mon, Apr 25, 2011 at 10:21:07PM +0200, Masklinn wrote:
> On 2011-04-25, at 22:13 , Brian Curtin wrote:
> > On Mon, Apr 25, 2011 at 15:05, Mike Graham <mikegraham at gmail.com> wrote:
> >> On Mon, Apr 25, 2011 at 3:51 PM, Brian Curtin <brian.curtin at gmail.com>
> >> wrote:
> >> [snipped]
> 
> With is actually a very nice name for some things, it creates very readable, english-looking code.
> 
> And what about `class`? Or `for` (that one clashes hard against the HTML object model, label elements have a for attribute). `in`, `except` or `is` may also be interesting in some cases.
> 
> Do all Python keywords have this issue? No, I doubt anybody's ever tried to called an attribute `elif`, but I definitely ran into the issue a few times.

for loops existed long before HTML, so I don't really see your point.
Again, I've never needed to use any of the reserved keywords for
variables.


From anikom15 at gmail.com  Fri Apr 29 00:33:44 2011
From: anikom15 at gmail.com (Westley =?iso-8859-1?Q?Mart=EDnez?=)
Date: Thu, 28 Apr 2011 15:33:44 -0700
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <4DB5CD47.3060808@interia.pl>
References: <4DB5CD47.3060808@interia.pl>
Message-ID: <20110428223344.GA2663@Smoke>

On Mon, Apr 25, 2011 at 09:36:39PM +0200, haael wrote:
> 
> Hello, guys.
> 
> I did post this idea a few months ago. Now the revised version.
> 
> 
> Goal:
> Let _all_ alphanumeric keywords be legal as names for variables,
> functions and classes, even the ones that are reserved words now.
> 
> Rationale:
> 1. Python took most good English words as reserved tokens. Situation
> goes worse from version to version. I often have hard time searching
> for acceptable synonyms.
> 2. Because of that, old Python programs cease to work, even if they
> do not use any abandoned features. Their only sin is using certain
> words that further versions of Python have stolen away.
> 3. Sometimes one needs to import keywords from some other language,
> XML be an example, or "translate" another programming language into
> Python in one way or another. Keyword reservation is a big problem
> then; it does not allow to use the natural Python syntax.
> 
> Solution:
> Let the parser treat all keywords that come after a dot (".") as
> regular identifiers.
> 
> 
> For attributes, nothing changes:
> > boo.for = 7
> 
> For names that are not attributes, only one syntax change is needed:
> let a dot precede any identifier.
> > .with = 3
> 
> Of course, if a keyword is not preceded by a dot, it would be
> treated as a reserved word, just like now.
> > with = 3  # syntax error
> 
> 
> There is only one case where a dot is used as a prefix of an
> identifier and that is a relative module import.
> > from .boo import foo
> My change is consistent with this case.
> 
> 
> One benefit would be that converting current programs to work with
> future versions would be a matter of simple grep.
> 
> Python is a great language. In my opinion, this change is the one
> last step to make it every geeky teenager's wet dream: the language
> where one can redefine almost anything. When I work with some
> problem, I always try to translate it to Python, solve and translate
> back. Prohibited identifier names are the main obstacle.
> 
> So, let's set the identifiers free and swallow all the world, making
> Python the least common denominator of every computer problem on
> this planet.
> 
> 
> Regards,
> Bartosz Tarnowski
> 
> 
> 
> 
>

I don't know about you guys but I've never needed a reserved word for a
variable name, and if I did, I'd just append an _ to it.


From tjreedy at udel.edu  Fri Apr 29 01:24:37 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 28 Apr 2011 19:24:37 -0400
Subject: [Python-ideas] [Python-Dev] the role of assert in the standard
	library ?
In-Reply-To: <BANLkTi=Ba-T8RHJ0e4_967qoc=65KsBXdw@mail.gmail.com>
References: <BANLkTimtoxmyNg9sn7e27soPBdxQMZ-q2Q@mail.gmail.com>	<ipb8qr$6jl$1@dough.gmane.org>
	<4DB94102.9020701@voidspace.org.uk>
	<BANLkTi=Ba-T8RHJ0e4_967qoc=65KsBXdw@mail.gmail.com>
Message-ID: <ipcsvk$t5l$1@dough.gmane.org>

On 4/28/2011 8:27 AM, Tarek Ziad? wrote:

 >> On 28/04/2011 09:34, Terry Reedy wrote:

 >>> My understanding is that assert can be used in production code but 
only to
 >>> catch logic errors by testing supposed invariants or postconditions. It
 >>> should not be used to test usage errors, including preconditions. 
In other
 >>> words, assert presence or absence should not affect behavior unless 
the code
 >>> has a bug.
 >
 > But it does affect the behaviour at the end: when the code has a bug,

That is just what I just said: 'unless the code has a bug'.

 > then the way the code works is affected and differs depending if -O is
 > used.

As long as Python has flags that affect code and its execution, the 
stdlib should be tested with them.

 > Let me take an example:
 >
 > bucket = []
 >
 > def add(stuff):
 >      bucket.append(stuff)
 >
 > def purge_and_do_something():
 >      bucket.clear()
 >      assert len(bucket) == 0
 >      ... do something by being sure the bucket is empty ...
 >
 >
 > here, we could say assert is legitimate, as it just checks a
 > post-condition.

No, not legitimate: the .clear() post-condition check should be in 
.clear(), not in code that uses it.

 > But this code is not thread-safe and if it's run via
 > several threads, some code could add something in the bucket while
 > purge() check for the assertion.

It is, or should be, well-known that concurrency, in general, makes 
programs non-deterministic (which is to say, potentially buggy relative 
to expectations). Purge needs a write-lock (that still allows .clear() 
to run).

 > My point is that I do not understand why there are assert calls to
 > check post-conditions.

Assert are inline unittests that long predate the x-unit modules for 
various languages. One advantage over separate test suites is that they 
test that the function works properly not only with the relatively few 
inputs in most test suites but also with all inputs in the integration 
and acceptance tests and, if left in place, in actual use.

 > Moreover, why -O and -OO are removing assertions in that case ?

A polynomial-time verification function may still take awhile.  For 
instance, is_sorted_version_of(input, output) has to verify both that 
output is sorted (easy) and that it is a permutation of the input 
(harder). So you might like it optionally gone in production but still 
present in source and optionally present in production where speed is 
not critical.

 > From what I understood so far, a line with an assert should be
 > considered as a dead code if we want the code to behave always the
 > same way. I remove dead code in my code...

I use asserts to test my custom test functions that they pass obviously 
good functions and fail intentionally bad functions with the proper 
error message. Not dead at all, especially when I 'refactor' (fiddle 
with) the test functin code. Of course, I could replace then with if (): 
raise or print, by why?

---
Terry Jan Reedy




From alexander.belopolsky at gmail.com  Fri Apr 29 01:26:15 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 28 Apr 2011 19:26:15 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
Message-ID: <BANLkTi=zX7MrqpdrkxF_DR2zAgnm+vAX-g@mail.gmail.com>

On Thu, Apr 28, 2011 at 6:10 PM, Guido van Rossum <guido at python.org> wrote:
..
> But isn't a similar change needed for Decimal?

I did not look into this, but decimal contexts allow for more
compliant implementations because you can trap FP exceptions
differently in different contexts.  We don't have this luxury with
floats.


From robert.kern at gmail.com  Fri Apr 29 03:38:13 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 28 Apr 2011 20:38:13 -0500
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>	<4DB97351.2050605@pearwood.info>	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>	<4DB98F1B.8060700@pearwood.info>	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
Message-ID: <ipd4q5$vog$1@dough.gmane.org>

On 4/28/11 5:10 PM, Guido van Rossum wrote:
> On Thu, Apr 28, 2011 at 12:25 PM, Alexander Belopolsky
> <alexander.belopolsky at gmail.com>  wrote:
>> I posted a patch implementing this proposal on the tracker:
>>
>> http://bugs.python.org/issue11949
>
> Interesting indeed! I'd like to hear from the numpy folks about this.

I'm personally -1, though mostly on general conservative principles. I'm sure 
there is some piece of code that will break, but I don't know how significant it 
would be.

I'm not sure that it solves a significant problem. I've never actually heard of 
anyone running into an infinite cycle due to NaNs, though a bit of Googling does 
suggest that it happens sometimes.

I don't think it really moves us closer to IEEE-754 compliance. The standard 
states (section 7. "Exceptions") "The default response to an exception shall be 
to proceed without a trap." Python only intermittently turns INVALID operations 
into exceptions, mostly just (-1.0)**0.5 and integer conversion (0/0.0 and x%0.0 
could be considered covered under the division by zero signal that *is* 
consistently turned into a Python exception). inf-inf, inf/inf, 0*inf, and 
inf%2.0, to give other examples of INVALID-signaling operations from the spec, 
all return a NaN without an exception. Given that we want to avoid exposing 
SIGFPE handlers for safety reasons, I think the status quo is a reasonable 
compromise interpretation of the spec.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From robert.kern at gmail.com  Fri Apr 29 03:40:15 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 28 Apr 2011 20:40:15 -0500
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTi=qd=UNcsrVL6cnx8pS4BXmUdKPFg@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<BANLkTi=qd=UNcsrVL6cnx8pS4BXmUdKPFg@mail.gmail.com>
Message-ID: <ipd4tv$vog$2@dough.gmane.org>

On 4/28/11 11:12 AM, Alexander Belopolsky wrote:

> The problem with faithfully implementing IEEE 754 in Python is that
> exceptions in IEEE standard don't have the same meaning as in Python.
>   IEEE 754 requires that a value is computed even when the operation
> signals an exception.  The program can then decide whether to
> terminate computation or propagate the value. In Python, we have to
> choose between raising an exception and returning the value.  We
> cannot have both.  It appears that in most cases IEEE 754 "INVALID"
> exception is treated as a terminating exception by Python and
> operations that signal INVALID in IEEE 754 raise an exception in
> Python.

This is not true. In fact, in most cases that issue an INVALID exception are 
passed silently in Python. See my response to Guido elsewhere in this thread for 
a nearly complete list.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From szport at gmail.com  Fri Apr 29 08:53:55 2011
From: szport at gmail.com (ZS)
Date: Fri, 29 Apr 2011 10:53:55 +0400
Subject: [Python-ideas] [[Python-Dev] PyObject_RichCompareBool identity
 shortcut] About specializing comparing objects in collections
Message-ID: <BANLkTinVAzyJkyYXc_icDOw+00atDM1YyA@mail.gmail.com>

Returning to the original topic post on the comparison of objects in
containers ...
Could be asked to enter a special method __equals__ of objects for
comparison in containers (by default, if it is not defined - use usual
method for compatibility), just as is done in C# (it's special
object's method Equals is used to compare items in the collection).

-----
Zaur


From ncoghlan at gmail.com  Fri Apr 29 08:55:31 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 29 Apr 2011 16:55:31 +1000
Subject: [Python-ideas] Make all keywords legal as an identifier
In-Reply-To: <20110428223344.GA2663@Smoke>
References: <4DB5CD47.3060808@interia.pl>
	<20110428223344.GA2663@Smoke>
Message-ID: <BANLkTik91+Z3aOHTgz2-qF4-yFC3fPDq0Q@mail.gmail.com>

On Fri, Apr 29, 2011 at 8:33 AM, Westley Mart?nez <anikom15 at gmail.com> wrote:
> I don't know about you guys but I've never needed a reserved word for a
> variable name, and if I did, I'd just append an _ to it.

"class" is the only one I have ever wanted to use, and "cls" is a
perfectly acceptable substitute in that case (other variants I have
seen are "klass" and "class_").

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Fri Apr 29 09:00:48 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 29 Apr 2011 17:00:48 +1000
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
Message-ID: <BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>

On Fri, Apr 29, 2011 at 8:10 AM, Guido van Rossum <guido at python.org> wrote:
> On Thu, Apr 28, 2011 at 12:25 PM, Alexander Belopolsky
> <alexander.belopolsky at gmail.com> wrote:
>> I posted a patch implementing this proposal on the tracker:
>>
>> http://bugs.python.org/issue11949
>
> Interesting indeed! I'd like to hear from the numpy folks about this.
>
> But isn't a similar change needed for Decimal?

decimal already works that way:

>>> from decimal import Decimal as d
>>> nan = d("nan")
>>> nan
Decimal('NaN')
>>> nan < 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python32\lib\decimal.py", line 887, in __lt__
    ans = self._compare_check_nans(other, context)
  File "C:\Python32\lib\decimal.py", line 788, in _compare_check_nans
    self)
  File "C:\Python32\lib\decimal.py", line 3926, in _raise_error
    raise error(explanation)
decimal.InvalidOperation: comparison involving NaN

+1 from me for making float NaNs raise ValueError for ordering comparisons.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From szport at gmail.com  Fri Apr 29 09:16:43 2011
From: szport at gmail.com (ZS)
Date: Fri, 29 Apr 2011 11:16:43 +0400
Subject: [Python-ideas] [[Python-Dev] PyObject_RichCompareBool identity
 shortcut] About specializing comparing objects in collections
Message-ID: <BANLkTimvVGVCpQumCxxBwfO9F6JODbD=2A@mail.gmail.com>

It was necessary to clarify the reason for the possible introduction
of a special method __equals__. The fact that the objects in python
are often interpreted as values. Method __eq__, IMHO is used more for
comparing objects as values??. Introduction of method __equals__ can
explicitly indicate the method of comparison of the objects as objects
and __eq__ objects as values??. If method __equals__ is not defined
then __eq__ will be used.

-- 
Zaur


From stephen at xemacs.org  Fri Apr 29 09:30:39 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 29 Apr 2011 16:30:39 +0900
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <4DB9932A.9070903@btinternet.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info> <4DB9932A.9070903@btinternet.com>
Message-ID: <87y62t8qg0.fsf@uwakimon.sk.tsukuba.ac.jp>

Rob Cliffe writes:

 > True.  But why be forced to walk on eggshells when writing a perfectly 
 > ordinary bit of code that "ought" to work as is?

What makes you think *anything* "ought" to "work" in the  presence of
NaNs?



From steve at pearwood.info  Fri Apr 29 10:03:35 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 29 Apr 2011 18:03:35 +1000
Subject: [Python-ideas] [[Python-Dev] PyObject_RichCompareBool identity
 shortcut] About specializing comparing objects in collections
In-Reply-To: <BANLkTinVAzyJkyYXc_icDOw+00atDM1YyA@mail.gmail.com>
References: <BANLkTinVAzyJkyYXc_icDOw+00atDM1YyA@mail.gmail.com>
Message-ID: <4DBA70D7.5020808@pearwood.info>

ZS wrote:
> Returning to the original topic post on the comparison of objects in
> containers ...
> Could be asked to enter a special method __equals__ of objects for
> comparison in containers (by default, if it is not defined - use usual
> method for compatibility), just as is done in C# (it's special
> object's method Equals is used to compare items in the collection).


I'm not sure I understand what that would gain us, have I missed something?

Currently, containers use identity tests as an optimization to avoid 
calling (potentially expensive) __eq__. What benefit do you have in mind 
to have containers call (potentially expensive) __equals__ instead of 
__eq__?

It might help if you tell us what you envisage float would have as 
__equals__ instead of the current __eq__.



-- 
Steven



From alexander.belopolsky at gmail.com  Fri Apr 29 17:57:14 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 29 Apr 2011 11:57:14 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <87y62t8qg0.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info> <4DB9932A.9070903@btinternet.com>
	<87y62t8qg0.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <BANLkTimQ1Fx2nfPrxpC96z-CcOfZJWfCHg@mail.gmail.com>

On Fri, Apr 29, 2011 at 3:30 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Rob Cliffe writes:
>
> ?> True. ?But why be forced to walk on eggshells when writing a perfectly
> ?> ordinary bit of code that "ought" to work as is?
>
> What makes you think *anything* "ought" to "work" in the ?presence of
> NaNs?

There are different shades of "not working".  In most cases, raising
an exception is preferable to silently producing garbage or entering
an infinite loop.  NaNs are unordered and NaN < 0 makes as much sense
as None < 0 or "abc" < 0.  The later operations raise an exception in
py3k.


From alexander.belopolsky at gmail.com  Fri Apr 29 18:14:02 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 29 Apr 2011 12:14:02 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>
Message-ID: <BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>

On Fri, Apr 29, 2011 at 3:00 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
..
> decimal already works that way:
>
>>>> from decimal import Decimal as d
>>>> nan = d("nan")
>>>> nan
> Decimal('NaN')
>>>> nan < 1
> ..
> decimal.InvalidOperation: comparison involving NaN

That's what I thought and contrary to what Robert said early in the
thread.  By default, decimal operations trap InvalidOperation,
DivisionByZero, and Overflow:

>>> decimal.getcontext()
Context(prec=28, rounding=ROUND_HALF_EVEN, Emin=-999999999,
Emax=999999999, capitals=1, clamp=0, flags=[],
traps=[InvalidOperation, DivisionByZero, Overflow])

The advantage that decimal has over float is that user can control
what is trapped:

>>> from decimal import *
>>> with localcontext(Context(traps=[])):
...      print(Decimal('NaN') < Decimal('0'))
...
False


From robert.kern at gmail.com  Fri Apr 29 18:28:03 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 29 Apr 2011 11:28:03 -0500
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>	<4DB97351.2050605@pearwood.info>	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>	<4DB98F1B.8060700@pearwood.info>	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>
	<BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>
Message-ID: <ipeouj$je1$1@dough.gmane.org>

On 4/29/11 11:14 AM, Alexander Belopolsky wrote:
> On Fri, Apr 29, 2011 at 3:00 AM, Nick Coghlan<ncoghlan at gmail.com>  wrote:
> ..
>> decimal already works that way:
>>
>>>>> from decimal import Decimal as d
>>>>> nan = d("nan")
>>>>> nan
>> Decimal('NaN')
>>>>> nan<  1
>> ..
>> decimal.InvalidOperation: comparison involving NaN
>
> That's what I thought and contrary to what Robert said early in the
> thread.

I have said nothing about decimal. I can requote the relevant portions of the 
IEEE-754 standard again, if you like.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From grosser.meister.morti at gmx.net  Fri Apr 29 18:37:46 2011
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Fri, 29 Apr 2011 18:37:46 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
Message-ID: <4DBAE95A.7000004@gmx.net>

I thought about this problem for I wile and I came up with this @decorator:


Usage:
------
 >>> @decorator
 >>> def my_deco(func,func_args,func_kwargs,deco_args...):
 >>> 	pass
 >>>
 >>> @my_deco(*deco_args,**deco_kwargs)
 >>> def func(*func_args,**func_kwargs):
 >>> 	pass

Or a more specific example:
 >>> @decorator
 >>> def deco(func,func_args,func_kwargs,a,b=12,**kwargs):
 >>> 	return (func(*func_args,**func_kwargs),func_args,func_kwargs,a,b,kwargs)
 >>>
 >>> @deco(1,f=12)
 >>> def foo(x,y,z=2,*args):
 >>> 	return (x,y,z,args)
 >>>
 >>> foo(5,6)
((5, 6, 2, ()), (5, 6, 2), {}, 1, 12, {'f': 12})

This fully supports *args and **kwargs besides regular arguments for the decorator and the decorated 
function. By that I mean func_args already contains the filled in default values of func as well as 
any regular arguments that where passed as keyword argument and argument passing errors are handled 
(e.g. passing an argument as positional and keyword argument).


Error handling example:
 >>> @deco(1,2,3)
 >>> def bar(x,y,z=2,*args):
 >>> 	return (x,y,z,args)
 >>>

Traceback (most recent call last):
   File "<pyshell#114>", line 1, in <module>
     @deco(1,2,3)
   File "<pyshell#96>", line 22, in _deco_deco
     deco_args, deco_kwargs = apply_deco_args(*deco_args, **deco_kwargs)
TypeError: deco() takes at most 2 arguments (3 given)

Or:
 >>> foo(5,6,y=33)

Traceback (most recent call last):
   File "<pyshell#112>", line 1, in <module>
     foo(5,6,y=33)
   File "<pyshell#96>", line 27, in _f
     func_args, func_kwargs = apply_func_args(*func_args, **func_kwargs)
TypeError: foo() got multiple values for keyword argument 'y'


Of course that always needs function call parenthesis on the decorator, even if the decorator does 
not take any arguments. Maybe it could be extended that in this case a more simple decorator 
mechanism is used. A decorator-decorator for decorators without arguments would be very simple (see 
end of mail).


Implementation:
---------------
from types import FunctionType, ClassType
from functools import wraps

def inspect_callable(func):
	"""-> (arg_names, co_flags, func_defaults, func_name)"""
	return _inspect_callable(func,set())

def _inspect_callable(func,visited):
	if func in visited:
		raise TypeError("'%s' object is not callable" % type(func).__name__)
	visited.add(func)
	if isinstance(func, FunctionType):
		co = func.func_code
		func_name = func.__name__
		arg_names = list(co.co_varnames[0:co.co_argcount])
		defaults = func.func_defaults
		flags = co.co_flags
	elif isinstance(func, ClassType):
		func_name = func.__name__
		arg_names, flags, defaults, member_name = _inspect_callable(func.__init__,visited)
		if arg_names:
			del arg_names[0]
	elif hasattr(func, '__call__'):
		func_name = '<%s object at 0x%x>' % (type(func).__name__, id(func))
		arg_names, flags, defaults, member_name = _inspect_callable(func.__call__,visited)
	else:
		raise TypeError("'%s' object is not callable" % type(func).__name__)
	return arg_names, flags, defaults, func_name

FUNC_ARGS   = 0x04
FUNC_KWARGS = 0x08
FUNC_GEN    = 0x20

# this function should probably be reimplemented in C:
def args_applyer(arg_names,flags=0,defaults=None,func_name=None):
	"""-> f(args..., [*varargs], [**kwargs]) -> ((args...)+varargs, kwargs)"""
	all_args = list(arg_names)
	if arg_names:
		body = ['(',','.join(arg_names),')']
	else:
		body = []
	if flags & FUNC_ARGS:
		args_name = '_args'
		i = 0
		while args_name in arg_names:
			args_name = '_args'+i
			i += 1
		all_args.append('*'+args_name)
		if arg_names:
			body.append('+')
		body.append(args_name)
	elif not arg_names:
		body.append('()')
	body.append(',')
	if flags & FUNC_KWARGS:
		kwargs_name = '_kwargs'
		i = 0
		while kwargs_name in arg_names:
			kwargs_name = '_kwargs'+i
			i += 1
		all_args.append('**'+kwargs_name)
		body.append(kwargs_name)
	else:
		body.append('{}')
	if func_name:
		apply_args = named_lambda(func_name,all_args,''.join(body))
	else:
		apply_args = eval('lambda %s: (%s)' % (','.join(all_args), ''.join(body)))
	if defaults:
		apply_args.func_defaults = defaults
	return apply_args

def named_lambda(name,args,body):
	code = 'def _named_lambda():\n\tdef %s(%s):\n\t\treturn %s\n\treturn %s' % (
		name, ','.join(args), body, name)
	del name, args, body
	exec(code)
	return _named_lambda()

# begin helper functions (not used by this module but might be handy for decorator developers)
def args_applyer_for(func):
	return args_applyer(*inspect_callable(func))

def apply_args(args,kwargs,arg_names,flags=0,defaults=None,func_name=None):
	return args_applyer(arg_names,flags,defaults,func_name)(*args,**kwargs)

def apply_args_for(func,args,kwargs):
	return args_applyer(*inspect_callable(func))(*args,**kwargs)
# end helper functions

def decorator(deco):
	"""deco(func,func_args,func_kwargs,deco_args...)
	
	@decorator
	def my_deco(func,func_args,func_kwargs,deco_args...):
		pass
	
	@my_deco(*deco_args,**deco_kwargs)
	def func(*func_args,**func_kwargs):
		pass
	"""
	arg_names, flags, defaults, deco_name = inspect_callable(deco)
	if flags & FUNC_ARGS == 0:
		if len(arg_names) < 3:
			raise TypeError('decorator functions need at least 3 ' +
				'arguments (func, func_args, func_kwargs)')
		del arg_names[0:3]
	apply_deco_args = args_applyer(arg_names,flags,defaults,deco_name)
	del flags, defaults
	@wraps(deco)
	def _deco_deco(*deco_args,**deco_kwargs):
		deco_args, deco_kwargs = apply_deco_args(*deco_args, **deco_kwargs)
		def _deco(func):
			apply_func_args = args_applyer(*inspect_callable(func))
			@wraps(func)
			def _f(*func_args,**func_kwargs):
				func_args, func_kwargs = apply_func_args(*func_args, **func_kwargs)
				return deco(func,func_args,func_kwargs,*deco_args,**deco_kwargs)
			return _f
		return _deco
	return _deco_deco

def simple_decorator(deco):
	"""deco(func,func_args,func_kwargs)
	
	@simple_decorator
	def my_deco(func,func_args,func_kwargs):
		pass
	
	@my_deco
	def func(*func_args,**func_kwargs):
		pass
	"""
	@wraps(deco)
	def _deco(func):
		apply_func_args = args_applyer(*inspect_callable(func))
		@wraps(func)
		def _f(*args,**kwargs):
			return deco(func,*apply_func_args(*args,**kwargs))
		return _f
	return _deco



From alexander.belopolsky at gmail.com  Fri Apr 29 18:39:55 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 29 Apr 2011 12:39:55 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <ipeouj$je1$1@dough.gmane.org>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>
	<BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>
	<ipeouj$je1$1@dough.gmane.org>
Message-ID: <BANLkTik-ugm2xOuV9PG_FYBu2U-kb-HrZg@mail.gmail.com>

On Fri, Apr 29, 2011 at 12:28 PM, Robert Kern <robert.kern at gmail.com> wrote:
..
> I have said nothing about decimal. I can requote the relevant portions of
> the IEEE-754 standard again, if you like.

Please do.  I had a draft of IEEE-754 standard somewhere at some
point, but not anymore.  I rely on Kahan's notes at
http://www.cs.berkeley.edu/~wkahan/ieee754status/ieee754.ps .


From grosser.meister.morti at gmx.net  Fri Apr 29 19:03:49 2011
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Fri, 29 Apr 2011 19:03:49 +0200
Subject: [Python-ideas] Proposal for new-style decorators
In-Reply-To: <4DBAE95A.7000004@gmx.net>
References: <BANLkTi=moxS+7OSO_kUn3S0s7xS0oxfXtQ@mail.gmail.com>
	<4DBAE95A.7000004@gmx.net>
Message-ID: <4DBAEF75.9020301@gmx.net>

Maybe even better, decorator and simple_decorator in one:


def decorator(deco):
	"""deco(func,func_args,func_kwargs,deco_args...)
	
	@decorator
	def my_deco(func,func_args,func_kwargs,deco_args...):
		pass
	
	@my_deco(*deco_args,**deco_kwargs)
	def func(*func_args,**func_kwargs):
		pass
		
	@decorator
	def my_deco2(func,func_args,func_kwargs):
		pass
		
	@my_deco2
	def func2(*func_args,**func_kwargs):
		pass

	@decorator
	def my_deco3(func,func_args,func_kwargs,x=1):
		pass
		
	@my_deco3()
	def func3(*func_args,**func_kwargs):
		pass
	"""
	arg_names, flags, defaults, deco_name = inspect_callable(deco)
	if flags & FUNC_ARGS == 0:
		if len(arg_names) < 3:
			raise TypeError('decorator functions need at least 3 ' +
				'arguments (func, func_args, func_kwargs)')
		del arg_names[0:3]
	if not arg_names and flags & (FUNC_ARGS | FUNC_KWARGS) == 0:
		# argument-less decorator
		del arg_names, flags, defaults, deco_name
		@wraps(deco)
		def _deco(func):
			apply_func_args = args_applyer(*inspect_callable(func))
			@wraps(func)
			def _f(*args,**kwargs):
				return deco(func,*apply_func_args(*args,**kwargs))
			return _f
		return _deco
	else:
		apply_deco_args = args_applyer(arg_names,flags,defaults,deco_name)
		del arg_names, flags, defaults, deco_name
		@wraps(deco)
		def _deco_deco(*deco_args,**deco_kwargs):
			deco_args, deco_kwargs = apply_deco_args(*deco_args, **deco_kwargs)
			def _deco(func):
				apply_func_args = args_applyer(*inspect_callable(func))
				@wraps(func)
				def _f(*func_args,**func_kwargs):
					func_args, func_kwargs = apply_func_args(
						*func_args, **func_kwargs)
					return deco(func,func_args,func_kwargs,
						*deco_args,**deco_kwargs)
				return _f
			return _deco
		return _deco_deco


From grosser.meister.morti at gmx.net  Fri Apr 29 19:33:01 2011
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Fri, 29 Apr 2011 19:33:01 +0200
Subject: [Python-ideas] a few decorator recipes
Message-ID: <4DBAF64D.30500@gmx.net>

A few decorator recipes that might be worthwhile to add to functools:

A way to assign annotations to functions/classes:

def annotations(**annots):
	def deco(obj):
		if hasattr(obj,'__annotations__'):
			obj.__annotations__.update(annots)
		else:
			obj.__annotations__ = annots
		return obj
	return deco

_NONE = object()
def getannot(obj, key, default=_NONE):
	if hasattr(obj, '__annotations__'):
		if default is _NONE:
			return obj.__annotations__[key]
		else:
			return obj.__annotations__.get(key, default)
	elif default is _NONE:
		raise KeyError(key)
	else:
		return default

def setannot(obj, key, value):
	if hasattr(obj, '__annotations__'):
		obj.__annotations__[key] = value
	else:
		obj.__annotations__ = {key: value}

Usage:
 >>> @annotations(foo='bar',egg='spam')
 >>> def foo():
 >>> 	pass
 >>>
 >>> getannot(foo, 'egg')
'spam'


A way to assign values to classes/functions (not of much use for classes, of course):

def assign(**values):
	def deco(obj):
		for key in values:
			setattr(obj, key, values[key])
		return obj
	return deco

Usage:
 >>> @assign(bla='bleh',x=12)
 >>> def foo():
 >>> 	pass
 >>>
 >>> foo.x
12


From stephen at xemacs.org  Fri Apr 29 20:47:19 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 30 Apr 2011 03:47:19 +0900
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTimQ1Fx2nfPrxpC96z-CcOfZJWfCHg@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info> <4DB9932A.9070903@btinternet.com>
	<87y62t8qg0.fsf@uwakimon.sk.tsukuba.ac.jp>
	<BANLkTimQ1Fx2nfPrxpC96z-CcOfZJWfCHg@mail.gmail.com>
Message-ID: <87r58k99oo.fsf@uwakimon.sk.tsukuba.ac.jp>

Alexander Belopolsky writes:
 > On Fri, Apr 29, 2011 at 3:30 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:

 > > Rob Cliffe writes:
 > >
 > > ?> True. ?But why be forced to walk on eggshells when writing a perfectly
 > > ?> ordinary bit of code that "ought" to work as is?
 > >
 > > What makes you think *anything* "ought" to "work" in the ?presence of
 > > NaNs?
 > 
 > There are different shades of "not working".  In most cases,
 > raising an exception is preferable to silently producing garbage or
 > entering an infinite loop.  NaNs are unordered and NaN < 0 makes as
 > much sense as None < 0 or "abc" < 0.  The later operations raise an
 > exception in py3k.

Sure, Python's behavior when asked to perform mathematical operations
that do not admit a usable definition can be improved, to the benefit
of people who write robust, high performance code.  I appreciate your
contribution to that discussion greatly.

But I really doubt that raising here is going to save anybody's
eggshells.  The cure you suggest might be better than silent garbage
or an infinite loop, but in production code you will still have to
think carefully about preventing or handling the exception.  Not to
mention finding a way to produce NaNs in the first place.  That's far
from what I would call "perfectly ordinary code working as is".



From robert.kern at gmail.com  Fri Apr 29 22:23:22 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 29 Apr 2011 15:23:22 -0500
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTik-ugm2xOuV9PG_FYBu2U-kb-HrZg@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>	<4DB97351.2050605@pearwood.info>	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>	<4DB98F1B.8060700@pearwood.info>	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>	<BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>	<ipeouj$je1$1@dough.gmane.org>
	<BANLkTik-ugm2xOuV9PG_FYBu2U-kb-HrZg@mail.gmail.com>
Message-ID: <ipf6nq$66v$1@dough.gmane.org>

On 4/29/11 11:39 AM, Alexander Belopolsky wrote:
> On Fri, Apr 29, 2011 at 12:28 PM, Robert Kern<robert.kern at gmail.com>  wrote:
> ..
>> I have said nothing about decimal. I can requote the relevant portions of
>> the IEEE-754 standard again, if you like.
>
> Please do.  I had a draft of IEEE-754 standard somewhere at some
> point, but not anymore.  I rely on Kahan's notes at
> http://www.cs.berkeley.edu/~wkahan/ieee754status/ieee754.ps .

(Section 7. "Exceptions") "The default response to an exception shall be to 
proceed without a trap."

IEEE-854 has the same sentence.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From alexander.belopolsky at gmail.com  Fri Apr 29 23:30:56 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 29 Apr 2011 17:30:56 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <ipf6nq$66v$1@dough.gmane.org>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>
	<BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>
	<ipeouj$je1$1@dough.gmane.org>
	<BANLkTik-ugm2xOuV9PG_FYBu2U-kb-HrZg@mail.gmail.com>
	<ipf6nq$66v$1@dough.gmane.org>
Message-ID: <BANLkTin7yMUH6b-awtBS8M7oKPO_JAJTAw@mail.gmail.com>

On Fri, Apr 29, 2011 at 4:23 PM, Robert Kern <robert.kern at gmail.com> wrote:
..
> (Section 7. "Exceptions") "The default response to an exception shall be to
> proceed without a trap."

I cannot find this phrase in my copy of IEEE Std 754-2008.  Instead, I
see the following in section 7.1:

"This clause also specifies default non-stop exception handling for
exception signals, which is to deliver a default result, continue
execution, and raise the corresponding status flag."

As I mentioned before, Python does not have a mechanism that would
allow to simultaneously raise an exception and deliver the result.  We
have to choose one or the other.  I think the choice made in the
decimal module is a reasonable one: trap Overflow, DivisionByZero, and
InvalidOperation while ignoring Underflow and Inexact.

The choices made for float operations are more ad-hoc: DivisionByZero
is always trapped:

>>> 1.0/0.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: float division by zero
>>> math.log(0.0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: math domain error

Overflow is trapped in some cases:

>>> 1e308 ** 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: (34, 'Result too large')

>>> math.exp(1e20)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: math range error

and ignored in others:

>>> 1e308+ 1e308
inf
>>> 1e200 * 1e200
inf

InvalidOperation is not handled consistently.  Let me copy the
relevant section of the standard and show Python's behavior for each
case where InvalidOperation exception is required by the standard:

"""
The invalid operation exception is signaled if and only if there is no
usefully definable result. In these cases the operands are invalid for
the operation to be performed.

For operations producing results in floating-point format, the default
result of an operation that signals the invalid operation exception
shall be a quiet NaN that should provide some diagnostic information
(see 6.2).

These operations are:

a)	any general-computational or signaling-computational operation on a
signaling NaN (see 6.2), except for some conversions (see 5.12)
"""

Python does not have support for sNaNs.  It is possible to produce a
float carrying an sNaN using struct.unpack, but the result behaves a
qNaN.  InvalidOperation not trapped.

"""
b)	multiplication: multiplication(0, ?) or multiplication(?, 0)
"""

>>> 0.0 * float('inf')
nan

InvalidOperation not trapped.

"""
c)	fusedMultiplyAdd: fusedMultiplyAdd(0, ?, c) or fusedMultiplyAdd(?,
0, c) unless c is a quiet NaN; if c is a quiet NaN then it is
implementation defined whether the invalid operation exception is
signaled
"""

Not applicable.  Python does not have fusedMultiplyAdd (x * y + z) function.

"""
d) addition or subtraction or fusedMultiplyAdd: magnitude subtraction
of infinities, such as: addition(+?, ??)
"""

>>> float('inf') + float('-inf')
nan

InvalidOperation not trapped.

"""
e)	division: division(0, 0) or division(?, ?)
"""

>>> 0.0/0.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: float division by zero

InvalidOperation trapped, but misreported as DivisionByZero.

>>> float('inf') / float('inf')
nan

"""
f)	remainder: remainder(x, y), when y is zero or x is infinite and
neither is NaN
"""

>>> 1.0 % 0.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: float modulo

InvalidOperation trapped, but misreported as DivisionByZero.

>>> float('inf') % 2.0
nan

InvalidOperation not trapped.

"""
g)	squareRoot if the operand is less than zero
"""

>>> math.sqrt(-1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: math domain error

InvalidOperation trapped.

"""
h)	quantize when the result does not fit in the destination format or
when one operand is finite and the other is infinite
"""

Not applicable.

"""
For operations producing no result in floating-point format, the
operations that signal the invalid operation exception are:
i)	conversion of a floating-point number to an integer format, when
the source is NaN, infinity, or a value that would convert to an
integer outside the range of the result format under the applicable
rounding attribute.
"""

>>> int(float('nan'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: cannot convert float NaN to integer

InvalidOperation trapped.

>>> int(float('inf'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: cannot convert float infinity to integer

InvalidOperation trapped, but misclassified as OverflowError.

"""
j)	comparison by way of unordered-signaling predicates listed in Table
5.2, when the operands are unordered
"""

This is the subject of my proposal.

>>> float('nan') < 0.0
False

InvalidOperation not trapped.

"""
k)	logB(NaN), logB(?), or logB(0) when logBFormat is an integer format
(see 5.3.3).
"""
Not applicable.


Overall, it appears that in cases where InvalidOperation was
anticipated, it was converted to some type of exception in Python.
Exceptions to this rule seem to be an accident of implementation.


From robert.kern at gmail.com  Sat Apr 30 00:11:54 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 29 Apr 2011 17:11:54 -0500
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTin7yMUH6b-awtBS8M7oKPO_JAJTAw@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>	<4DB97351.2050605@pearwood.info>	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>	<4DB98F1B.8060700@pearwood.info>	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>	<BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>	<ipeouj$je1$1@dough.gmane.org>	<BANLkTik-ugm2xOuV9PG_FYBu2U-kb-HrZg@mail.gmail.com>	<ipf6nq$66v$1@dough.gmane.org>
	<BANLkTin7yMUH6b-awtBS8M7oKPO_JAJTAw@mail.gmail.com>
Message-ID: <ipfd3b$86o$1@dough.gmane.org>

On 4/29/11 4:30 PM, Alexander Belopolsky wrote:
> On Fri, Apr 29, 2011 at 4:23 PM, Robert Kern<robert.kern at gmail.com>  wrote:
> ..
>> (Section 7. "Exceptions") "The default response to an exception shall be to
>> proceed without a trap."
>
> I cannot find this phrase in my copy of IEEE Std 754-2008.  Instead, I
> see the following in section 7.1:
>
> "This clause also specifies default non-stop exception handling for
> exception signals, which is to deliver a default result, continue
> execution, and raise the corresponding status flag."

Ah. I have the 1985 version.

> Overall, it appears that in cases where InvalidOperation was
> anticipated, it was converted to some type of exception in Python.
> Exceptions to this rule seem to be an accident of implementation.

Well, for comparisons at least, it seems to have been anticipated, and returning 
a value was intentional. From the comments documenting float_richcompare() in 
floatobject.c:

/* Comparison is pretty much a nightmare.  When comparing float to float,
  * we do it as straightforwardly (and long-windedly) as conceivable, so
  * that, e.g., Python x == y delivers the same result as the platform
  * C x == y when x and/or y is a NaN.
...

I'm not sure there's any evidence that the other behaviors have not been 
anticipated or are accidents of implementation. The ambiguous inf operations are 
documented and doctested in Lib/test/ieee754.txt.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From alexander.belopolsky at gmail.com  Sat Apr 30 00:36:41 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 29 Apr 2011 18:36:41 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <ipfd3b$86o$1@dough.gmane.org>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>
	<BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>
	<ipeouj$je1$1@dough.gmane.org>
	<BANLkTik-ugm2xOuV9PG_FYBu2U-kb-HrZg@mail.gmail.com>
	<ipf6nq$66v$1@dough.gmane.org>
	<BANLkTin7yMUH6b-awtBS8M7oKPO_JAJTAw@mail.gmail.com>
	<ipfd3b$86o$1@dough.gmane.org>
Message-ID: <BANLkTincs6re0Xe4NG6zns8MbeuC8B=ZbQ@mail.gmail.com>

On Fri, Apr 29, 2011 at 6:11 PM, Robert Kern <robert.kern at gmail.com> wrote:
..
> Well, for comparisons at least, it seems to have been anticipated, and
> returning a value was intentional. From the comments documenting
> float_richcompare() in floatobject.c:
>
> /* Comparison is pretty much a nightmare. ?When comparing float to float,
> ?* we do it as straightforwardly (and long-windedly) as conceivable, so
> ?* that, e.g., Python x == y delivers the same result as the platform
> ?* C x == y when x and/or y is a NaN.
> ?

Well, I may be overly pedantic, but this comment only mentions that
the author considered == comparison and not ordering of NaNs.   I
guess we need to ask Tim Peters if he considered the fact that x < NaN
is invalid operation according to IEEE 754 while x == NaN is not.
Tim?


From alexander.belopolsky at gmail.com  Sat Apr 30 00:50:05 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 29 Apr 2011 18:50:05 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTin7yMUH6b-awtBS8M7oKPO_JAJTAw@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>
	<BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>
	<ipeouj$je1$1@dough.gmane.org>
	<BANLkTik-ugm2xOuV9PG_FYBu2U-kb-HrZg@mail.gmail.com>
	<ipf6nq$66v$1@dough.gmane.org>
	<BANLkTin7yMUH6b-awtBS8M7oKPO_JAJTAw@mail.gmail.com>
Message-ID: <BANLkTik4sc3Yv6sbXoiKheUN6NKBq3sZXA@mail.gmail.com>

On Fri, Apr 29, 2011 at 5:30 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
..
> As I mentioned before, Python does not have a mechanism that would
> allow to simultaneously raise an exception and deliver the result. ?We
> have to choose one or the other.

I made this argument several times and it went unchallenged, but I now
realize that Python does have a mechanism that would allow to
simultaneously raise an exception and deliver the result.  This is
what warnings do.  Since changing NaN < 0 to raise an error would have
to be done by issuing a deprecation warning first, why can't we just
issue appropriate warning on invalid operations?  Isn't this what
numpy does in some cases?


From robert.kern at gmail.com  Sat Apr 30 01:02:03 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 29 Apr 2011 18:02:03 -0500
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTik4sc3Yv6sbXoiKheUN6NKBq3sZXA@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>	<4DB97351.2050605@pearwood.info>	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>	<4DB98F1B.8060700@pearwood.info>	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>	<BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>	<ipeouj$je1$1@dough.gmane.org>	<BANLkTik-ugm2xOuV9PG_FYBu2U-kb-HrZg@mail.gmail.com>	<ipf6nq$66v$1@dough.gmane.org>	<BANLkTin7yMUH6b-awtBS8M7oKPO_JAJTAw@mail.gmail.com>
	<BANLkTik4sc3Yv6sbXoiKheUN6NKBq3sZXA@mail.gmail.com>
Message-ID: <ipfg1c$m1n$1@dough.gmane.org>

On 4/29/11 5:50 PM, Alexander Belopolsky wrote:
> On Fri, Apr 29, 2011 at 5:30 PM, Alexander Belopolsky
> <alexander.belopolsky at gmail.com>  wrote:
> ..
>> As I mentioned before, Python does not have a mechanism that would
>> allow to simultaneously raise an exception and deliver the result.  We
>> have to choose one or the other.
>
> I made this argument several times and it went unchallenged, but I now
> realize that Python does have a mechanism that would allow to
> simultaneously raise an exception and deliver the result.  This is
> what warnings do.  Since changing NaN<  0 to raise an error would have
> to be done by issuing a deprecation warning first, why can't we just
> issue appropriate warning on invalid operations?  Isn't this what
> numpy does in some cases?

We have a configurable mechanism that lets you change between ignoring, warning, 
and raising an exception (and a few others).

[~]
|1> with np.errstate(invalid='raise'):
..>     np.array([np.inf]) / np.array([np.inf])
..>
---------------------------------------------------------------------------
FloatingPointError                        Traceback (most recent call last)
/Users/rkern/<ipython-input-1-d0b8f36f6dea> in <module>()
       1 with np.errstate(invalid='raise'):
----> 2     np.array([np.inf]) / np.array([np.inf])
       3

FloatingPointError: invalid value encountered in divide

[~]
|2> with np.errstate(invalid='ignore'):
..>     np.array([np.inf]) / np.array([np.inf])
..>

[~]
|3> with np.errstate(invalid='warn'):
..>     np.array([np.inf]) / np.array([np.inf])
..>
/Library/Frameworks/Python.framework/Versions/Current/bin/ipython:2: 
RuntimeWarning: invalid value encountered in divide


I think I could support issuing a warning. Beats the hell out of arguing over 
fine details of ancient standards intended for low-level languages and hardware. :-)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From tjreedy at udel.edu  Sat Apr 30 02:37:28 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 29 Apr 2011 20:37:28 -0400
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTik4sc3Yv6sbXoiKheUN6NKBq3sZXA@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>	<4DB97351.2050605@pearwood.info>	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>	<4DB98F1B.8060700@pearwood.info>	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>	<BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>	<ipeouj$je1$1@dough.gmane.org>	<BANLkTik-ugm2xOuV9PG_FYBu2U-kb-HrZg@mail.gmail.com>	<ipf6nq$66v$1@dough.gmane.org>	<BANLkTin7yMUH6b-awtBS8M7oKPO_JAJTAw@mail.gmail.com>
	<BANLkTik4sc3Yv6sbXoiKheUN6NKBq3sZXA@mail.gmail.com>
Message-ID: <ipflk8$emj$1@dough.gmane.org>

On 4/29/2011 6:50 PM, Alexander Belopolsky wrote:

> realize that Python does have a mechanism that would allow to
> simultaneously raise an exception and deliver the result.  This is
> what warnings do.  Since changing NaN<  0 to raise an error would have
> to be done by issuing a deprecation warning first, why can't we just
> issue appropriate warning on invalid operations?

Adding FloatWarning would probably be easier than changing behavior to 
raise an exception. People who care could change into an exception.

-- 
Terry Jan Reedy



From stephen at xemacs.org  Sat Apr 30 07:35:32 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 30 Apr 2011 14:35:32 +0900
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <ipflk8$emj$1@dough.gmane.org>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>
	<BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>
	<ipeouj$je1$1@dough.gmane.org>
	<BANLkTik-ugm2xOuV9PG_FYBu2U-kb-HrZg@mail.gmail.com>
	<ipf6nq$66v$1@dough.gmane.org>
	<BANLkTin7yMUH6b-awtBS8M7oKPO_JAJTAw@mail.gmail.com>
	<BANLkTik4sc3Yv6sbXoiKheUN6NKBq3sZXA@mail.gmail.com>
	<ipflk8$emj$1@dough.gmane.org>
Message-ID: <87d3k48fob.fsf@uwakimon.sk.tsukuba.ac.jp>

Terry Reedy writes:
 > On 4/29/2011 6:50 PM, Alexander Belopolsky wrote:
 > 
 > > realize that Python does have a mechanism that would allow to
 > > simultaneously raise an exception and deliver the result.  This is
 > > what warnings do.  Since changing NaN<  0 to raise an error would have
 > > to be done by issuing a deprecation warning first, why can't we just
 > > issue appropriate warning on invalid operations?
 > 
 > Adding FloatWarning would probably be easier than changing behavior to 
 > raise an exception. People who care could change into an exception.

+1  That looks both formally compliant and much better than allowing
what is often a programming error to fail silently.




From dickinsm at gmail.com  Sat Apr 30 10:13:46 2011
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sat, 30 Apr 2011 09:13:46 +0100
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTi=qd=UNcsrVL6cnx8pS4BXmUdKPFg@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<BANLkTi=qd=UNcsrVL6cnx8pS4BXmUdKPFg@mail.gmail.com>
Message-ID: <BANLkTik78t6bxAnwYui-fHBRVjTCtx+vuA@mail.gmail.com>

On Thu, Apr 28, 2011 at 5:12 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> Furthermore, IEEE 754 specifies exactly what I propose:
>
> """
> IEEE 754 assigns values to all relational expressions involving NaN.
> In the syntax of C , the predicate x != y is True but all others, x <
> y , x <= y , x == y , x >= y and x > y, are False whenever x or y or
> both are NaN, and then all but x != y and x == y are INVALID
> operations too and must so signal.
> """
> -- Lecture Notes on the Status of IEEE Standard 754 for Binary
> Floating-Point Arithmetic by Prof. W. Kahan
> http://www.cs.berkeley.edu/~wkahan/ieee754status/ieee754.ps

Note that this text refers to the obsolete IEEE 754-1985, not the
current version of the standard.

IEEE 754 isn't really much help here:  the current version of the
standard specifies (in section 5.11: Details of comparison predicates)
*twenty-two* distinct comparison predicates.  That includes, for
example:

  'compareSignalingGreater'

which is a greater-than comparison that signals an invalid operation
exception on a comparison involving NaNs.  But it also includes:

  'compareQuietGreater'

which returns False for comparisons involving NaNs.  And IEEE 754 has
nothing to say about how the specified operations should be mapped to
language constructs---that's out of scope for the specification.  (It
does happen to list plain '>' as one of the names for
'compareSignalingGreater', but I don't think it's realistic to try to
read anything into that.)

I'm -0 on the proposal: I don't think there's enough of a real problem
here to justify the change.

Mark


From dickinsm at gmail.com  Sat Apr 30 10:23:01 2011
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sat, 30 Apr 2011 09:23:01 +0100
Subject: [Python-ideas] Disallow orderring comparison to NaN
In-Reply-To: <BANLkTin7yMUH6b-awtBS8M7oKPO_JAJTAw@mail.gmail.com>
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
	<BANLkTi=CmE=NtsHmD_ttN2heXh7emAz2yA@mail.gmail.com>
	<BANLkTi=tEjCV7Z5ANh9r1Q++-oH7sT-5nw@mail.gmail.com>
	<ipeouj$je1$1@dough.gmane.org>
	<BANLkTik-ugm2xOuV9PG_FYBu2U-kb-HrZg@mail.gmail.com>
	<ipf6nq$66v$1@dough.gmane.org>
	<BANLkTin7yMUH6b-awtBS8M7oKPO_JAJTAw@mail.gmail.com>
Message-ID: <BANLkTikXHors0nJrMFOx-QjyMMNJ3ibL9A@mail.gmail.com>

On Fri, Apr 29, 2011 at 10:30 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> The choices made for float operations are more ad-hoc: DivisionByZero
> is always trapped:
> [...]

Roughly, the current situation is that math module operations try to
consistently follow IEEE 754 exceptions:  an IEEE 754 overflow is
converted to an OverflowError, while invalid-operation or
divide-by-zero signals produce a Python ValueError.

Basic arithmetic is another story:  ** behaves more-or-less like the
math module operations, but the arithmetic operations mainly produce
nans or infinities, except that division by zero is trapped.

IMO, the ideal (ignoring backwards compatibility) would be to have
OverflowError / ZeroDivisionError / ValueError produced wherever
IEEE754 says that overflow / divide-by-zero / invalid-operation should
be signaled.

Mark


From solipsis at pitrou.net  Sat Apr 30 12:30:55 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 30 Apr 2011 12:30:55 +0200
Subject: [Python-ideas] Disallow orderring comparison to NaN
References: <BANLkTimBeVuk7r5dYA0LfR=Bn4w_fsmoJw@mail.gmail.com>
	<4DB97351.2050605@pearwood.info>
	<BANLkTikVaoxmxujBxEeHuEdrmLwqDAsY+g@mail.gmail.com>
	<4DB98F1B.8060700@pearwood.info>
	<BANLkTi=fEL=U=QA1sypa3tC-G-LhONq03g@mail.gmail.com>
	<BANLkTimNi4-mXKaPOgKwkPgi4Er4Bv360w@mail.gmail.com>
	<BANLkTi=f2-2_660TuFy683YJbF6PsB_mdA@mail.gmail.com>
	<ipd4q5$vog$1@dough.gmane.org>
Message-ID: <20110430123055.4ffd7083@pitrou.net>

On Thu, 28 Apr 2011 20:38:13 -0500
Robert Kern <robert.kern at gmail.com> wrote:
> On 4/28/11 5:10 PM, Guido van Rossum wrote:
> > On Thu, Apr 28, 2011 at 12:25 PM, Alexander Belopolsky
> > <alexander.belopolsky at gmail.com>  wrote:
> >> I posted a patch implementing this proposal on the tracker:
> >>
> >> http://bugs.python.org/issue11949
> >
> > Interesting indeed! I'd like to hear from the numpy folks about this.
> 
> I'm personally -1, though mostly on general conservative principles. I'm sure 
> there is some piece of code that will break, but I don't know how significant it 
> would be.
> 
> I'm not sure that it solves a significant problem. I've never actually heard of 
> anyone running into an infinite cycle due to NaNs, though a bit of Googling does 
> suggest that it happens sometimes.

Same as Robert. This does not seem very useful and may break existing
code.

It also opens the door for attacks against code which takes floats as
input strings and parses them using the float() constructor. An
attacker can pass "nan", which will be converted successfully and can
later raise an exception at an arbitrary point. Applications will have
to actively protect against this, which is an unnecessary nuisance.

Regards

Antoine.




From benjamin at python.org  Sat Apr 30 21:17:58 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 30 Apr 2011 19:17:58 +0000 (UTC)
Subject: [Python-ideas] a few decorator recipes
References: <4DBAF64D.30500@gmx.net>
Message-ID: <loom.20110430T211534-378@post.gmane.org>

Mathias Panzenb?ck <grosser.meister.morti at ...> writes:
> 
> def annotations(**annots):
> 	def deco(obj):
> 		if hasattr(obj,'__annotations__'):
> 			obj.__annotations__.update(annots)
> 		else:
> 			obj.__annotations__ = annots
> 		return obj
> 	return deco

Why would you want to do that?

> 
> def setannot(obj, key, value):

I don't see the point.