[Python-checkins] python/nondist/peps pep-0343.txt,1.16,1.17
gvanrossum@users.sourceforge.net
gvanrossum at users.sourceforge.net
Wed Jun 1 17:13:46 CEST 2005
Update of /cvsroot/python/python/nondist/peps
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv22078
Modified Files:
pep-0343.txt
Log Message:
Specify generator enhancements. Change keyword to 'with'.
Index: pep-0343.txt
===================================================================
RCS file: /cvsroot/python/python/nondist/peps/pep-0343.txt,v
retrieving revision 1.16
retrieving revision 1.17
diff -u -d -r1.16 -r1.17
--- pep-0343.txt 31 May 2005 20:27:15 -0000 1.16
+++ pep-0343.txt 1 Jun 2005 15:13:37 -0000 1.17
@@ -1,5 +1,5 @@
PEP: 343
-Title: Anonymous Block Redux
+Title: Anonymous Block Redux and Generator Enhancements
Version: $Revision$
Last-Modified: $Date$
Author: Guido van Rossum
@@ -11,17 +11,13 @@
Introduction
- After a lot of discussion about PEP 340 and alternatives, I've
- decided to withdraw PEP 340 and propose a slight variant on
- PEP 310.
-
-Evolutionary Note
-
- After ample discussion on python-dev, I'll add back a mechanism
+ After a lot of discussion about PEP 340 and alternatives, I
+ decided to withdraw PEP 340 and proposed a slight variant on PEP
+ 310. After more discussion, I have added back a mechanism
for raising an exception in a suspended generator using a throw()
method, and a close() method which throws a new GeneratorExit
- exception. Until I get a chance to update the PEP, see reference
- [2]. I'm also leaning towards 'with' as the keyword.
+ exception; these additions were first proposed in [2] and
+ universally approved of. I'm also changing the keyword to 'with'.
Motivation and Summary
@@ -53,7 +49,7 @@
with VAR = EXPR:
BLOCK
- which roughly translates into
+ which roughly translates into this:
VAR = EXPR
VAR.__enter__()
@@ -74,7 +70,7 @@
goto (a break, continue or return), BLOCK2 is *not* reached. The
magic added by the with-statement at the end doesn't affect this.
- (You may ask, what if a bug in the __exit__ method causes an
+ (You may ask, what if a bug in the __exit__() method causes an
exception? Then all is lost -- but this is no worse than with
other exceptions; the nature of exceptions is that they can happen
*anywhere*, and you just have to live with that. Even if you
@@ -89,16 +85,18 @@
Inspired by a counter-proposal to PEP 340 by Phillip Eby I tried
to create a decorator that would turn a suitable generator into an
- object with the necessary __entry__ and __exit__ methods. Here I
- ran into a snag: while it wasn't too hard for the locking example,
- it was impossible to do this for the opening example. The idea
- was to define the template like this:
+ object with the necessary __enter__() and __exit__() methods.
+ Here I ran into a snag: while it wasn't too hard for the locking
+ example, it was impossible to do this for the opening example.
+ The idea was to define the template like this:
@with_template
def opening(filename):
f = open(filename)
- yield f
- f.close()
+ try:
+ yield f
+ finally:
+ f.close()
and used it like this:
@@ -106,21 +104,21 @@
...read data from f...
The problem is that in PEP 310, the result of calling EXPR is
- assigned directly to VAR, and then VAR's __exit__ method is called
- upon exit from BLOCK1. But here, VAR clearly needs to receive the
- opened file, and that would mean that __exit__ would have to be a
- method on the file.
+ assigned directly to VAR, and then VAR's __exit__() method is
+ called upon exit from BLOCK1. But here, VAR clearly needs to
+ receive the opened file, and that would mean that __exit__() would
+ have to be a method on the file.
While this can be solved using a proxy class, this is awkward and
made me realize that a slightly different translation would make
writing the desired decorator a piece of cake: let VAR receive the
- result from calling the __enter__ method, and save the value of
- EXPR to call its __exit__ method later. Then the decorator can
- return an instance of a wrapper class whose __enter__ method calls
- the generator's next() method and returns whatever next() returns;
- the wrapper instance's __exit__ method calls next() again but
- expects it to raise StopIteration. (Details below in the section
- Optional Generator Decorator.)
+ result from calling the __enter__() method, and save the value of
+ EXPR to call its __exit__() method later. Then the decorator can
+ return an instance of a wrapper class whose __enter__() method
+ calls the generator's next() method and returns whatever next()
+ returns; the wrapper instance's __exit__() method calls next()
+ again but expects it to raise StopIteration. (Details below in
+ the section Optional Generator Decorator.)
So now the final hurdle was that the PEP 310 syntax:
@@ -128,41 +126,61 @@
BLOCK1
would be deceptive, since VAR does *not* receive the value of
- EXPR. Given PEP 340, it was an easy step to:
+ EXPR. Borrowing from PEP 340, it was an easy step to:
with EXPR as VAR:
BLOCK1
- or, using an alternate keyword that has been proposed a number of
- times:
+ Additional discussion showed that people really liked being able
+ to "see" the exception in the generator, even if it was only to
+ log it; the generator is not allowed to yield another value, since
+ the with-statement should not be usable as a loop (raising a
+ different exception is marginally acceptable). To enable this, a
+ new throw() method for generators is proposed, which takes three
+ arguments representing an exception in the usual fashion (type,
+ value, traceback) and raises it at the point where the generator
+ is suspended.
- do EXPR as VAR:
- BLOCK1
+ Once we have this, it is a small step to proposing another
+ generator method, close(), which calls throw() with a special
+ exception, GeneratorExit. This tells the generator to exit, and
+ from there it's another small step to proposing that close() be
+ called automatically when the generator is garbage-collected.
+
+ Then, finally, we can allow a yield-statement inside a try-finally
+ statement, since we can now guarantee that the finally-clause will
+ (eventually) be executed. The usual cautions about finalization
+ apply -- the process may be terminated abruptly without finalizing
+ any objects, and objects may be kept alive forever by cycles or
+ memory leaks in the application (as opposed to cycles or leaks in
+ the Python implementation, which are taken care of by GC).
+
+ Note that we're not guaranteeing that the finally-clause is
+ executed immediately after the generator object becomes unused,
+ even though this is how it will work in CPython. This is similar
+ to auto-closing files: while a reference-counting implementation
+ like CPython deallocates an object as soon as the last reference
+ to it goes away, implementations that use other GC algorithms do
+ not make the same guarantee. This applies to Jython, IronPython,
+ and probably to Python running on Parrot.
Use Cases
See the Examples section near the end.
-Specification
+Specification: The 'with' Statement
A new statement is proposed with the syntax:
- do EXPR as VAR:
+ with EXPR as VAR:
BLOCK
- Here, 'do' and 'as' are new keywords; EXPR is an arbitrary
+ Here, 'with' and 'as' are new keywords; EXPR is an arbitrary
expression (but not an expression-list) and VAR is an arbitrary
assignment target (which may be a comma-separated list).
The "as VAR" part is optional.
- The choice of the 'do' keyword is provisional; an alternative
- under consideration is 'with'.
-
- A yield-statement is illegal inside BLOCK. This is because the
- do-statement is translated into a try/finally statement, and yield
- is illegal in a try/finally statement.
-
The translation of the above statement is:
abc = EXPR
@@ -209,94 +227,184 @@
non-local goto should be considered unexceptional for the purposes
of a database transaction roll-back decision.
-Optional Generator Decorator
+Specification: Generator Enhancements
+
+ Let a generator object be the iterator produced by calling a
+ generator function. Below, 'g' always refers to a generator
+ object.
+
+ New syntax: yield allowed inside try-finally
+
+ The syntax for generator functions is extended to allow a
+ yield-statement inside a try-finally statement.
+
+ New generator method: throw(type, value, traceback)
+
+ g.throw(type, value, traceback) causes the specified exception to
+ be thrown at the point where the generator g is currently
+ suspended (i.e. at a yield-statement, or at the start of its
+ function body if next() has not been called yet). If the
+ generator catches the exception and yields another value, that is
+ the return value of g.throw(). If it doesn't catch the exception,
+ the throw() appears to raise the same exception passed it (it
+ "falls through"). If the generator raises another exception (this
+ includes the StopIteration produced when it returns) that
+ exception is raised by the throw() call. In summary, throw()
+ behaves like next() except it raises an exception at the
+ suspension point. If the generator is already in the closed
+ state, throw() just raises the exception it was passed without
+ executing any of the generator's code.
+
+ The effect of raising the exception is exactly as if the
+ statement:
+
+ raise type, value, traceback
+
+ was executed at the suspension point. The type argument should
+ not be None.
+
+ New standard exception: GeneratorExit
+
+ A new standard exception is defined, GeneratorExit, inheriting
+ from Exception. A generator should handle this by re-raising it
+ or by raising StopIteration.
+
+ New generator method: close()
+
+ g.close() is defined by the following pseudo-code:
+
+ def close(self):
+ try:
+ self.throw(GeneratorExit, GeneratorExit(), None)
+ except (GeneratorExit, StopIteration):
+ pass
+ else:
+ raise TypeError("generator ignored GeneratorExit")
+ # Other exceptions are not caught
+
+ New generator method: __del__()
+
+ g.__del__() is an alias for g.close(). This will be called when
+ the generator object is garbage-collected (in CPython, this is
+ when its reference count goes to zero). If close() raises an
+ exception, a traceback for the exception is printed to sys.stderr
+ and further ignored; it is not propagated back to the place that
+ triggered the garbage collection. This is consistent with the
+ handling of exceptions in __del__() methods on class instances.
+
+ If the generator object participates in a cycle, g.__del__() may
+ not be called. This is the behavior of CPython's current garbage
+ collector. The reason for the restriction is that the GC code
+ needs to "break" a cycle at an arbitrary point in order to collect
+ it, and from then on no Python code should be allowed to see the
+ objects that formed the cycle, as they may be in an invalid state.
+ Objects "hanging off" a cycle are not subject to this restriction.
+ Note that it is unlikely to see a generator object participate in
+ a cycle in practice. However, storing a generator object in a
+ global variable creates a cycle via the generator frame's
+ f_globals pointer. Another way to create a cycle would be to
+ store a reference to the generator object in a data structure that
+ is passed to the generator as an argument. Neither of these cases
+ are very likely given the typical pattern of generator use.
+
+Generator Decorator
It is possible to write a decorator that makes it possible to use
- a generator that yields exactly once to control a do-statement.
+ a generator that yields exactly once to control a with-statement.
Here's a sketch of such a decorator:
class Wrapper(object):
+
def __init__(self, gen):
self.gen = gen
- self.state = "initial"
+
def __enter__(self):
- assert self.state == "initial"
- self.state = "entered"
try:
return self.gen.next()
except StopIteration:
- self.state = "error"
- raise RuntimeError("template generator didn't yield")
- def __exit__(self, *args):
- assert self.state == "entered"
- self.state = "exited"
- try:
- self.gen.next()
- except StopIteration:
- return
+ raise RuntimeError("generator didn't yield")
+
+ def __exit__(self, type, value, traceback):
+ if type is None:
+ try:
+ self.gen.next()
+ except StopIteration:
+ return
+ else:
+ raise RuntimeError("generator didn't stop")
else:
- self.state = "error"
- raise RuntimeError("template generator didn't stop")
+ try:
+ self.gen.throw(type, value, traceback)
+ except (type, StopIteration):
+ return
+ else:
+ raise RuntimeError("generator caught exception")
- def do_template(func):
+ def with_template(func):
def helper(*args, **kwds):
return Wrapper(func(*args, **kwds))
return helper
This decorator could be used as follows:
- @do_template
+ @with_template
def opening(filename):
f = open(filename) # IOError here is untouched by Wrapper
yield f
f.close() # Ditto for errors here (however unlikely)
- A robust implementation of such a decorator should be made part of
- the standard library.
+ A robust implementation of this decorator should be made part of
+ the standard library, but not necessarily as a built-in function.
+ (I'm not sure which exception it should raise for errors;
+ RuntimeError is used above as an example only.)
-Other Optional Extensions
+Optional Extensions
It would be possible to endow certain objects, like files,
- sockets, and locks, with __enter__ and __exit__ methods so that
- instead of writing:
+ sockets, and locks, with __enter__() and __exit__() methods so
+ that instead of writing:
- do locking(myLock):
+ with locking(myLock):
BLOCK
one could write simply:
- do myLock:
+ with myLock:
BLOCK
I think we should be careful with this; it could lead to mistakes
like:
f = open(filename)
- do f:
+ with f:
BLOCK1
- do f:
+ with f:
BLOCK2
- which does not do what one might think (f is closed when BLOCK2 is
- entered).
+ which does not do what one might think (f is closed before BLOCK2
+ is entered).
+
+ OTOH such mistakes are easily diagnosed.
Examples
- Several of these examples contain "yield None". If PEP 342 is
- accepted, these can be changed to just "yield".
+ (Note: several of these examples contain "yield None". If PEP 342
+ is accepted, these can be changed to just "yield".)
1. A template for ensuring that a lock, acquired at the start of a
block, is released when the block is left:
- @do_template
+ @with_template
def locking(lock):
lock.acquire()
- yield None
- lock.release()
+ try:
+ yield None
+ finally:
+ lock.release()
Used as follows:
- do locking(myLock):
+ with locking(myLock):
# Code here executes with myLock held. The lock is
# guaranteed to be released when the block is left (even
# if via return or by an uncaught exception).
@@ -304,29 +412,32 @@
2. A template for opening a file that ensures the file is closed
when the block is left:
- @do_template
+ @with_template
def opening(filename, mode="r"):
f = open(filename, mode)
- yield f
- f.close()
+ try:
+ yield f
+ finally:
+ f.close()
Used as follows:
- do opening("/etc/passwd") as f:
+ with opening("/etc/passwd") as f:
for line in f:
print line.rstrip()
3. A template for committing or rolling back a database
transaction; this is written as a class rather than as a
- decorator since it requires access to the exception information:
+ decorator since it requires access to the exception
+ information:
class transactional:
def __init__(self, db):
self.db = db
def __enter__(self):
self.db.begin()
- def __exit__(self, *args):
- if args and args[0] is not None:
+ def __exit__(self, type, value, tb):
+ if type is not None:
self.db.rollback()
else:
self.db.commit()
@@ -338,47 +449,51 @@
self.lock = lock
def __enter__(self):
self.lock.acquire()
- def __exit__(self, *args):
+ def __exit__(self, type, value, tb):
self.lock.release()
(This example is easily modified to implement the other
- examples; it shows how much simpler generators are for the same
- purpose.)
+ examples; it shows the relative advantage of using a generator
+ template.)
5. Redirect stdout temporarily:
- @do_template
+ @with_template
def redirecting_stdout(new_stdout):
save_stdout = sys.stdout
sys.stdout = new_stdout
- yield None
- sys.stdout = save_stdout
+ try:
+ yield None
+ finally:
+ sys.stdout = save_stdout
Used as follows:
- do opening(filename, "w") as f:
- do redirecting_stdout(f):
+ with opening(filename, "w") as f:
+ with redirecting_stdout(f):
print "Hello world"
This isn't thread-safe, of course, but neither is doing this
- same dance manually. In a single-threaded program (e.g., a
- script) it is a totally fine way of doing things.
+ same dance manually. In single-threaded programs (for example,
+ in scripts) it is a popular way of doing things.
6. A variant on opening() that also returns an error condition:
- @do_template
+ @with_template
def opening_w_error(filename, mode="r"):
try:
f = open(filename, mode)
except IOError, err:
yield None, err
else:
- yield f, None
- f.close()
+ try:
+ yield f, None
+ finally:
+ f.close()
Used as follows:
- do opening_w_error("/etc/passwd", "a") as f, err:
+ with opening_w_error("/etc/passwd", "a") as f, err:
if err:
print "IOError:", err
else:
@@ -389,7 +504,7 @@
import signal
- do signal.blocking():
+ with signal.blocking():
# code executed without worrying about signals
An optional argument might be a list of signals to be blocked;
@@ -401,19 +516,21 @@
import decimal
- @do_template
- def with_extra_precision(places=2):
+ @with_template
+ def extra_precision(places=2):
c = decimal.getcontext()
saved_prec = c.prec
c.prec += places
- yield None
- c.prec = saved_prec
+ try:
+ yield None
+ finally:
+ c.prec = saved_prec
Sample usage (adapted from the Python Library Reference):
def sin(x):
"Return the sine of x as measured in radians."
- do with_extra_precision():
+ with extra_precision():
i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1
while s != lasts:
lasts = s
@@ -423,31 +540,33 @@
sign *= -1
s += num / fact * sign
# The "+s" rounds back to the original precision,
- # so this must be outside the do-statement:
+ # so this must be outside the with-statement:
return +s
9. Here's a more general Decimal-context-switching template:
- @do_template
- def with_decimal_context(newctx=None):
+ @with_template
+ def decimal_context(newctx=None):
oldctx = decimal.getcontext()
if newctx is None:
newctx = oldctx.copy()
decimal.setcontext(newctx)
- yield newctx
- decimal.setcontext(oldctx)
+ try:
+ yield newctx
+ finally:
+ decimal.setcontext(oldctx)
- Sample usage (adapted from the previous one):
+ Sample usage:
def sin(x):
- do with_decimal_context() as ctx:
+ with decimal_context() as ctx:
ctx.prec += 2
- # Rest of algorithm the same
+ # Rest of algorithm the same as above
return +s
- (Nick Coghlan has proposed to add __enter__ and __exit__
+ (Nick Coghlan has proposed to add __enter__() and __exit__()
methods to the decimal.Context class so that this example can
- be simplified to "do decimal.getcontext() as ctx: ...".)
+ be simplified to "with decimal.getcontext() as ctx: ...".)
References
More information about the Python-checkins
mailing list