[Python-Dev] code blocks using 'for' loops and generators

[Josiah Carlson]

Brian Sabbey <sabbey at u.washington.edu> wrote:
> 
> On Sat, 12 Mar 2005, Josiah Carlson wrote:
> 
> > I say it is magical.  Why?  The way you propose it, 'yield' becomes an
> > infix operator, with the name provided on the left being assigned a
> > value produced by something that isn't explicitly called, referenced, or
> > otherwise, by the right.  In fact, I would say, it would be akin to the
> > calling code modifying gen.gi_frame._f_locals directly.  Such "action at
> > a distance", from what I understand, is wholly frowned upon in Python.
> > There also does not exist any other infix operator that does such a
> > thing (=, +=, ... assign values, but where the data comes from is
> > obvious).
> >
> > Bad things to me (so far):
> > 1. Assignment is not obvious.
> > 2. Where data comes from is not obvious.
> > 3. Action at a distance like nothing else in Python.
> > 4. No non-'=' operator assigns to a local namespace.
> 
> If it is too much action at a distance, one could also use the syntax "a = 
> yield b", as has been suggested before, but I preferred it without the 
> '='.  With the '=', I don't see how it is any more action at a distance 
> than "a = yield_func(b)" or "for i in f()".  In all of these cases, values 
> are being passed when the scope is changing.

Right, but as I said before, when using any other assignment operator,
where the value comes from is obvious: the call or statement offered to
the right of the operator.  In your example, the data comes from the
caller, and even if you were to offer documentation, it is not obvious
who or what the caller is, or whether or not they would actually provide
such a value (what would happen if they didn't?).

Essentially what you are wanting to do is to take what would normally be
multiple function calls and place them in a generator to package up setup,
body, finalization.

l = load_pickle(filename) #setup
l.append(...)             #body
write_pickle(filename, l) #finalization

The above is /the/ canonical way to do this in basically every
programming language.  Sticking it in a generator for a sense of being
'clean' is preposterous.


> If you want to argue that both of these syntaxes are too ugly to be 
> allowed, then ok.

I stand by my original "ick".


> > Or you could even use a class instance.
> >
> > class foo:
> >    def pickled_file(self, name):
> >        f = open(name, 'r')
> >        yield pickle.load(f)
> >        f.close()
> >        f = open(name, 'w')
> >        pickle.dump(self.l, f)
> >        f.close()
> >
> > fi = foo()
> > for l in fi.pickled_file('greetings.pickle'):
> >     l.append('hello')
> >     l.append('howdy')
> >     fi.l = l
> >
> > If you can call the function, there is always a shared namespace.  It
> > may not be obvious (you may need to place the function as a method of an
> > instance), but it is still there.
> 
> The disadvantage of this method is that it is not clear where the self.l 
> values is coming from just by reading the generator method definition. If 
> it is coming from the code block, why not be able to explicity write it 
> that way?  Similarly, "continue l" explicitly shows that you are passing a 
> value back to the generator.

External to pickled_file, it is explicit where your 'l' or 'self.l' thing
is coming from.  You hit the nail on the head, the problem with /both/
your syntax and the equivalent that I offer above is that inside
pickled_file, it is not obvious where self.l (or equivalently the 'l' in
"l yield...") is coming from.  Why?  Because pulling the body out of a
setup/body/finalization series messes that up.

You would be better off either calling 3 functions like I provided above
(saving people the confusion of "what the hell is this for loop doing
here?"), or putting everything in one function/method, with a callback
like...

def pickled_file(filename, callback):
    #setup...
    callback(args) #body
    #finalization

> > The point of your proposed syntax is to inject data back into a
> > generator from an external namespace.  Right?
> 
> The point is to inject data back into a generator in a clean, explicit 
> way;  I understand there are other ways in which one can already do this.

But it is neither explicit nor clean.  Explicit from the outside,
perhaps, but not inside your generator.  And it certainly is not clean
because it pisses all over the history of "don't to anything too magical"
that Python seems to have strived for.

What if I were to tell you that "a and b" would start assigning a value
to "a" that wasn't "b" or "a", wasn't anything even related to "a" or "b",
and would be produced by code that isn't even referenced by "a", "b", or
anything else in the local namespace?  You'd probably tell me that I'd
had a few too many drinks, and you'd probably be right.  You are
essentially asking for "yield" to behave like this, and I think you've
had a few too many drinks *wink*.


Steven has pointed out PEP 288, and in my opinion, PEP 288 is the right
thing to do in regards to passing exceptions to generators.  Use the
'throw' method throw an exception, perfect.  It solves a long-standing
problem, and if /you/ want to use it to pass values, you can!

class Data(Exception):
    pass

def foo(filename):
    ...
    try:
        yield ...
    except Data, data:
        l = data
    ...

Assuming PEP 288 were in.  Yes, the above smells of a hack, but it is
obvious that the value assigned to 'l' is coming from an exception that
needs to be triggered by either the statement on the right side of the
yield, or offered via the gen.throw() method.  I personally would not
condone such use of gen.throw(), but I also don't condone the use of
"eval" and "exec", and that hasn't stopped thousands from doing so.


> > Furthermore, I would say that using yield semantics in the way that you
> > are proposing, while being discussed in other locations (the IBM article
> > on cooperative multithreading via generators springs to mind), is a
> > clever hack, but not something that should be supported via syntax.
> 
> My impression is that being able to use yield in this way is one of the 
> primary reasons for the success of ruby.  Therefore, I do not see it as 
> just a clever hack.  It is a clean, explicity way to pass values when 
> changing scope.

Ah, but Ruby is not Python, Ruby is Ruby.  Python only recently got
generators; before then we saved state explicitly (thank you Guido and
everyone else who made generators happen).  As such, while it may be
"old hat" to Ruby, it is relatively new to Python.

I stand by my "clever hack" statement, and I will agree to disagree with
you on it, like I agree to disagree with you about both the necessity of
passing arbitrary values back into a generator (I think it is poor style,
and too much brain overhead to wonder where data is coming from), and
the necessity of a new syntax to make such things "easier" (if (ab)using
generators in such a way makes /anything/ "easier").


 - Josiah