[Python-Dev] exec/with thunk-handling proposal

[Moore, Paul]

holger krekel wrote:
> Michael Hudson wrote:
> > holger krekel <pyth@devel.trillke.net> writes:
> > > I think we can may get away with only a "weak" keyword
> > > and allow the aforementioned encapsulation of execution=20
> > > events into an object like this:
> > >
> > >     exec expr [with params]: suite
> >=20
> > Gut reaction: ugh!
>=20
> then i guess you rather prefer new keywords.  I thought that there
> is a general reluctance to introduce new keywords *and* many
> people dislike 'exec' for its existence.  So reusing it
> (and we are dealing with execution aspects IMO) makes
> some sense to me.

I can see your point. Personally, I don't have any great problems with =
"exec",
so I'd like to understand better how your proposal integrates this new =
use of
exec with the existing use. And at some point, it will be necessary to =
confirm
that the parser can be made to distinguish the various uses (it seems to =
me
that extra lookahead will be needed).

> > > where the expression is evaluated to return a
> > > "thunk" handler with these optional "execution" hooks:
> > >
> > >     def __enter__(self): =20
> > >         "before suite start"
> > >
> > >     def __except__(self, type, value, tb):=20
> > >         "swallow given exception, reraise if neccessary"
> > >
> > >     def __leave__(self):
> > >         """upon suite finish (not called if __except__=20
> > >            exists and an exception happened)
> > >         """
> > >
> > > The above "with" parameters (of the form name=3Dexpr, =
comma-separated)=20
> > > are bound in local (or global/nested) *and* handler instance=20
> > > namespace.  The 'suite' is what we call "thunk".
> > >
> > > The above logic allows clean timely finalization for
> > > *multiple* ressources:
> > >
> > >     exec autoclose() with f1=3Dopen(name1), f2=3Dopen(name2, 'w'):
> > >         for line in f1:
> > >             ...
> > >             f2.write(...)
> >=20
> > That looks messy.
>=20
> as compared to?  I think i have seen messier ideas with all kinds
> of brackets and double-colons :-)

I agree with you here. This looks OK to me in terms of syntax. I'm not =
so sure
of the semantics - that may be the bit Michael thought was "messy". =
Binding
the with parameters in two places feels odd, and in particular injecting =
the
parameter names into the instance namespace is unusual (I know of no =
other
construct in Python which implicitly changes an instance namespace like =
this).

And as far as the local namespace is concerned, consider

    exec something() with f1=3D12:
        suite
    # Out of the exec now - what is the value of f1???

I assume that the assignment to f1 is still in existence, otherwise =
you're
inventing a new way of defining a scope. But if the binding to f1 does =
still
exist, then that looks odd, because "exec ... with f1=3D12" *reads* as =
if f1 is
only in scope for the duration of the statement. You seem to lose either
way...

> > > which would execute as follows
> > >
> > >     a) autoclose() instance is created and stored as the=20
> > >        "thunk"-handler
> >=20
> > We need a name for these.  I've been using "monitor" for a while, =
but
> > I'm not sure it's that apposite.
>=20
> So far i'd liked "execution handler" because 'enter/exit/except' are =
IMHO
> execution events which you can hook into.  Do you agree with calling
> them 'execution events' or how would you call them?=20

The way you've defined things, I see the logic - it's hooks into the =
execution
model. I'm not sure I'd have thought of that way of looking at it for =
myself,
though...

> > >     b) f1/f2 are stored as attributes on the autoclose instance
> > >
> > >     c) f1/f2 are put into the local/global namespace (and nested =
ones
> > >        if rebinding is allowed)
> > >
> > >     d) thunk executes (for line ...)
> > >
> > >     e) autoclose 'leave' hook is called (with or without =
exception)
> > >        and is implemented like this:
> > >        =20
> > >        def __leave__(self):
> > >             for obj in self.__dict__.values():=20
> > >                 obj.close()

But what if the object was more complex, and needed its own state? How =
could
it tell which were "its own" variables and which had been injected? =
(Other
than by hard coding a list of "its own" variables to ignore, which is =
utterly
horrible).

> > >     f) thunk handler is removed=20
> >=20
> > "Too much magic!"
>=20
> Hmmm. That seems a bit unfair to me as many other proposals didn't =
care=20
> to elaborate the exact sequence.   You probably are refering to the=20
> "namespace interactions" as the other stuff is probably the same
> with any other proposal so far.

No, I agree with Michael here. Just because other proposals handwave and =
don't
give proper semantics doesn't mean that they don't also have too much =
magic.
In my book "unclear semantics" is a worse accusation than "too much =
magic".
But both are bad :-)

> > > Because computing 'f1' may succeed but 'f2' can subsequently
> > > fail the assignments *have to* execute within "autoclose"
> > > control. =20

Hmm. In the equivalent try...finally construct, the idiom is to open the =
file
*outside* the try statement, so that exceptions when opening the file =
don't
get accidentally caught. You need to address this - particularly in the =
case
of multiple variables.

> Would the following be the correct way to achieve this with your =
patch?=20
>=20
>     f1=3Dopen(inputfn)
>     with autoclose(f1):
>         f2 =3D open(outputfn, 'w')
>         with autoclose(f2):
>             for line in f1:
>                 ...
>                 f2.write(line)
>=20
> I think there should be a better solution for multiple ressources.

In many ways, I agree with you. I originally wanted multiple =
expressions, and
assignments within the "with" statement. But people argued against them =
(look
back over the thread if you want the details). I honestly don't believe =
that
the with statement will ever please everyone - the best we can hope for =
is to
let it handle the simple cases well, with clean, well-defined semantics =
with
as few surprises as possible, and leave the remaining cases to the =
existing
try statement (or to a more ambitious thunk-based mechanism).

There is a real risk of slimming the with statement down to uselessness =
by
this philosophy. But this may simply indicate that the need is perceived
rather than real :-)

> > > Now on to the usage of the except hook.  Nice use cases might be
> > >
> > >     exec retry(maxretry=3D3, on=3DIOError):=20
> > >         # do network io-stuff
> > > or
> > >     exec skip_on(AttributeError, TypeError):
> > >         some_object.notify_hook()
> > >
> > > but i am sure there are more.  Exception handling is often
> > > ugly when inlined with the code.  I think that stating=20
> > > 'exception behaviour' up-front allows to write nice=20
> > > readable constructs.=20
> >=20
> > I am *still* not convinced that an __except__ hook is worth the =
pain.
> > Can you implement those for me?
>=20
> Maybe, i haven't looked at your implementation yet and there are
> some other projects pending :-)

There's no implementation of any form of except hook in existence yet
(excluding the IEXEC patch). The key point here is that your explanation =
of
semantics above didn't cover __except__ hooks. I'd like to see =
__except__
integrated into the semantics you describe, and then with that as a =
basis I's
like to see definitions of retry() and skip_on().

The idea looks interesting, but I'd like to see concrete code.

> > yields already can't go in blocks with finally statements, right?
>=20
> right.=20
>=20
> > Would you propose calling the __enter__ method each time the =
generator
> > resumed?
>=20
> yes, neccessary for locking (as the above example tried to indicate).

But not for files. It's arguable that you need __enter__, __exit__, =
__yield__,
and __resume__ hooks for full generality.

On the other hand, YAGNI makes more sense to me...

> > > If there is interest i can probably modify my patch=20
> > > to allow all of the proposed syntax so that you could=20
> > > play around with it. =20

Working code for *any* of these ideas could only be a good thing.

User code which benefits from the new features is also good. The =
auto-closing
file example has been worked to death, and its main effect for me so far =
has
been to make me wonder whether we need *any* of these mechanisms :-) The =
lock
example is better, but still pretty borderline. I'd really, really like =
to see
a substantial, non-toy use case.

Actually, my main experience is C/C++ based, where blocks with many =
memory
allocations and the occasional open file is the canonical example. =
Freeing and
closing in reverse order, under all error conditions, is a *pain*. But =
for
this, the C++ "resource acquisition is initialisation" idiom works
brilliantly. It's a shame it can't be made to work in Python.

Paul.