[Python-Dev] code blocks using 'for' loops and generators

[Josiah Carlson]

Brian Sabbey <sabbey at u.washington.edu> wrote:
> 
> On Fri, 11 Mar 2005, Josiah Carlson wrote:
> 
> > My first reaction to the proposal was "ick".  Why?  Because every time
> > I've seen a mechanism for modifying the internals of generators
> > constructed using yield, the syntax has been awful; in my opinion
> > (whether my opinion means anything is another matter), the syntax you
> > propose is quite awful,
> 
> I find it quite natural.  Stuff on the right of 'yield' is going out, 
> stuff on the left is coming in.  Since 'yield' is different than return in 
> that it marks a spot at which execution both leaves and re-enters the 
> frame, it makes sense that 'yield' should have a syntax that indicates as 
> much.

Inventors always find their ideas to be quite natural; if they weren't,
they wouldn't have come to them.


> > and the functionality you desire does not
> > require syntax modification to be possible.
> 
> Was the functionality provided by generators or decorators or anything 
> else impossible before those were introduced?  Of course not.  The point 
> is to make things easier and more natural, not to enable some previously 
> impossible functionality.

I don't find the syntax you provide to be 'natural'.  Trying to convince
me of the 'naturalness' of it is probably not going to be productive,
but as I said in my original email, "whether my opinion means anything
is another matter".

Being that no one else has publically responded to your post, and it has
been nearly 24 hours, my feeling is that either there are more important
things going on (Python 2.4.1, sum/product/any/all discussion, etc.),
people haven't been paying attention in recent days (due to the higher
volume), people are not sufficiently one side or another to comment, or
some other thing.


> > Strawman 1: the syntax
> >
> > def pickled_file(name):
> >     f = open(name, 'r')
> >     l yield pickle.load(f)
> >     ^
> > ------
> > |    f.close()
> > |    f = open(name, 'w')
> > |    pickle.dump(l, f)
> > |    f.close()
> > |
> > While this is currently a syntax error, it is not clear that an
> > assignment is happening /at all/, and I don't believe it would be /even
> > if/ if were documented, and every time someone opened up Python it said
> > "This is an assignment!" with an example. It looks too magical to me
> > (and from a guy who had previously proposed 'Some' as the opposite of
> > 'None', that's saying something).
> 
> Perhaps you are right, I don't know.  It seems to me that most people 
> would have to see the syntax once to know exactly what is going on, but I 
> certainly don't know that for sure.  Either way, I'd hate to have all my 
> suggestions dismissed because of the syntax of this one piece.


I say it is magical.  Why?  The way you propose it, 'yield' becomes an
infix operator, with the name provided on the left being assigned a
value produced by something that isn't explicitly called, referenced, or
otherwise, by the right.  In fact, I would say, it would be akin to the
calling code modifying gen.gi_frame._f_locals directly.  Such "action at
a distance", from what I understand, is wholly frowned upon in Python.
There also does not exist any other infix operator that does such a
thing (=, +=, ... assign values, but where the data comes from is
obvious).

Bad things to me (so far):
1. Assignment is not obvious.
2. Where data comes from is not obvious.
3. Action at a distance like nothing else in Python.
4. No non-'=' operator assigns to a local namespace.


> > Strawman 2: putting data back into a generator
> >
> > def pickled_file(name):
> >     f = open(name, 'r')
> >     yield pickle.load(f)
> >     f.close()
> >     f = open(name, 'w')
> >     pickle.dump(l, f)
> >     f.close()
> >
> > Keep your code, except toss that leading 'l'.
> >
> > for l in pickled_file('greetings.pickle'):
> >     l.append('hello')
> >     l.append('howdy')
> >
> > Toss that 'continue l', and your code will work (as long as both the
> > function and the for are sitting in the same namespace).
> 
> But they're *not* in the same namespace necessarily.  That is entirely the 
> point.  One is changing scope but has no clean way to pass values.  How is 
> making 'l' some (more) global variable possibly a clearer way to pass it 
> to the generator?  Your argument is like saying one does not need to 
> return values from a function because we could always just use a global 
> variable to do it.

Or you could even use a class instance.

class foo:
    def pickled_file(self, name):
        f = open(name, 'r')
        yield pickle.load(f)
        f.close()
        f = open(name, 'w')
        pickle.dump(self.l, f)
        f.close()

fi = foo()
for l in fi.pickled_file('greetings.pickle'):
     l.append('hello')
     l.append('howdy')
     fi.l = l

If you can call the function, there is always a shared namespace.  It
may not be obvious (you may need to place the function as a method of an
instance), but it is still there.


> > Hrm, not good enough?  Use a Queue, or use another variable in a
> > namespace accessable to both your function and your loop.
> 
> Again, I entirely realize it's possible to do these things now, but that 
> is not the point.

The point of your proposed syntax is to inject data back into a
generator from an external namespace.  Right?

My point is that if you are writing software, there are already fairly
reasonable methods to insert data back into a generator from an external
namespace.

Furthermore, I would say that using yield semantics in the way that you
are proposing, while being discussed in other locations (the IBM article
on cooperative multithreading via generators springs to mind), is a
clever hack, but not something that should be supported via syntax.

Considering the magical nature of how you propose to change yield, I
can't see any serious developer of the Python language (those with
commit access to Python CVS) saying anything other than "-1".  Coupled
with the fact that I cannot recall Guido or anyone else here ever having
said "it would be nice if we could put stuff back into generators", my
feeling is that your syntax proposal is not going to make it (I could be
wrong).


 - Josiah