[Python-Dev] Possible context managers in stdlib

[Nick Coghlan]

Skip Montanaro wrote:
> After seeing so many messages about "with" statements my eyes began to glaze
> over, so I stopped following that thread.  Then I saw mention of "context
> manager" with no reference to any PEPs or to the with statement to provide
> context.

The main outcome of the PEP 343 terminology discussion was some proposed 
documentation I put on the Sourceforge patch tracker ([1]). The patch is 
currently assigned to Raymond (since he started the terminology discussion) 
but any other reviews would be welcome.

Since SF currently doesn't want to play, and the proposed documentation isn't 
that long, I've included the latest version below for anyone who wants to read it.

> None of the context-providing messages seemed to have been indexed
> by Google when I checked, so searching for "Python context manager" failed
> to return anything useful.  Hence the post.

Google appears to have spidered the list archives some time today, so anyone 
else doing the same search should get some relevant hits.

Cheers,
Nick.

[1] http://www.python.org/sf/1234057
==========================================
With Statements and Context Management

A frequent need in programming is to ensure a particular action is
taken after a specific section of code has been executed (such as
closing a file or releasing a lock). Traditionally, this is handled
using 'try'/'finally' statements. However, that approach can lead
to the reproduction of non-trivial amounts of boilerplate whenever
the action needs to be invoked.

A simpler way to achieve this in Python is to use the 'with' statement
along with the appropriate context manager. Context managers define an
action which is taken to enter the context and a second action to
exit the context (usually restoring the environment that existed
before the context was entered). The 'with' statement ensures that the
context is entered and exited at the appropriate times (that is,
before and after the execution of the suite contained in the 'with'
statement).

The precise behaviour of the 'with' statement is governed by the
supplied context manager - an object which supports the context
management protocol. This protocol consists of two methods:

      __enter__(self):
        Context managers use this method to enter the desired context
      before the execution of the contained suite.
        This method is called without arguments before execution of
      the contained suite starts. If the 'as' clause of the 'with'
      statement is used, the value returned from this method is
      assigned to the specified target.
        Many context managers will return self from this method, but
      returning a different object may make sense for some managers
      (for example, see the 'closing' suite manager described below).

      __exit__(self, exc_type, exc_value, exc_traceback):
        Context managers use this method to exit the context after
      execution of the contained suite.
        This method is called after execution of the contained suite
      is completed. If execution completed due to an exception, the
      details of that exception are passed as arguments. Otherwise,
      all three arguments are set to None.
        If exception details are passed in, and this method returns
      without incident, then the original exception continues to
      propagate. Otherwise, the exception raised by this method will
      replace the original exception.


Using Contexts to Manage Resources

The simplest use of context management is to strictly control the
handling of key resources (such as files, generators, database
connections, synchronisation locks).

These resource managers will generally acquire the resource in their
__enter__ method, although some resource managers may accept the
resource to be managed as an argument to the constructor or acquire it
during construction. Resource managers will then release the resource
in their __exit__ method.

For example, the following context manager allows prompt closure of
any resource with a 'close' method (e.g. a generator or file):

      class closing(object):
          def __init__(self, resource):
              self.resource = resource

          def __enter__(self):
              return self.resource

          def __exit__(self, *exc_info):
              self.resource.close()


     with closing(my_generator()) as g:
         # my_generator() is assigned to g via call to __enter__()
         for item in g:
             print item
     # g is closed as the with statement ends

Some resources (such as threading.Lock) support the context management
protocol natively, allowing them to be used directly in 'with'
statements. The meaning of the established context will depend on the
specific resource. In the case of threading.Lock, the lock is acquired
by the __enter__ method, and released by the __exit__ method.

   with the_lock:
       # Suite is executed with the_lock held
   # the_lock is released as the with statement ends


More Context Management Examples

While resource management may be the most obvious use of the context
management protocol, many more uses are possible (otherwise it would
have been called the resource management protocol!).

For example, when used as a context manager, a decimal.Context object
will set itself as the current Decimal arithmetic context in the
__enter__ method, and then automatically revert back to the previous
Decimal arithmetic context in the __exit__ method. This allows the
code in the contained suite to manipulate the Decimal arithmetic
context freely, without needing to worry about manually undoing any
changes.

     with decimal.getcontext() as ctx:
         ctx.prec = 48
         # Perform high precision calculations within the context
     # Precision reverts to default here

Another example is the use of contexts to handle insertion of the
appropriate start and end tags when generating HTML:

     class tag(object):
         def __init__(self,name):
             self.name = name

         def __enter__(self):
             print "<%s>" % self.name

         def __exit__(self, exc_type, *exc_details):
             if exc_type is None:
                 print "</%s>" % self.name


     with tag('html'):
        with tag('body'):
           with tag('h1'):
              print "Some heading"
           with tag('p'):
              print "This is paragraph 1"
           with tag('p'):
              print "This is paragraph 2"
           with tag('h2'):
              print "Another heading"

Some other possibilities for context management include automatic
exception logging and handling of database transactions.


Using Generators to Define Context Managers

In conjunction with the 'contextmanager' decorator, Python's
generators provide a convenient way to implement the context
management protocol, and share state between the __enter__ and
__exit__ methods.

The generator must yield exactly once during normal execution. The
context manager's __enter__ method executes the generator up to that
point, and the value yielded is returned. The remainder of the
generator is executed by the context manager's __exit__ method. Any
exceptions that occur in the managed context will be injected into the
generator at the location of the yield statement.

For example, here are the 'closing' and 'tag' context manager examples
written using generators:

      @contextmanager
      def closing(resource):
          try:
              yield resource
          finally:
              resource.close()


      @contextmanager
      def tag(name):
          print "<%s>" % name
          yield None
          print "</%s>" % name

The operation of the contextmanager decorator is described by the
following Python equivalent (although the exact error messages may
differ):

      class ContextWrapper(object):
          def __init__(self, gen):
              self.gen = gen

          def __enter__(self):
              try:
                  return self.gen.next()
              except StopIteration:
                  raise RuntimeError("generator didn't yield")

          def __exit__(self, type, value, traceback):
              if type is None:
                  try:
                      self.gen.next()
                  except StopIteration:
                      return
                  else:
                      raise RuntimeError("generator didn't stop")
              else:
                  try:
                      self.gen.throw(type, value, traceback)
                  except (type, StopIteration):
                      return
                  else:
                      raise RuntimeError("generator didn't stop")


      def contextmanager(func):
          def factory(*args, **kwds):
              return ContextWrapper(func(*args, **kwds))
          factory.__name__ = func.__name__
          factory.__doc__ = func.__doc__
          factory.__dict__ = func.__dict__
          return factory
==========================================



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://boredomandlaziness.blogspot.com