A "scopeguard" for Python

[Alf P. Steinbach]

* Robert Kern:
> On 2010-03-04 17:52 , Alf P. Steinbach wrote:
>> * Robert Kern:
>>> On 2010-03-04 12:37 PM, Alf P. Steinbach wrote:
>>>> * Robert Kern:
>>>>> On 2010-03-04 10:56 AM, Alf P. Steinbach wrote:
>>>>>> * Robert Kern:
>>>>>>> On 2010-03-03 18:49 PM, Alf P. Steinbach wrote:
>>>> [snippety]
>>>>>>>
>>>>>>>> If you call the possibly failing operation "A", then that 
>>>>>>>> systematic
>>>>>>>> approach goes like this: if A fails, then it has cleaned up its own
>>>>>>>> mess, but if A succeeds, then it's the responsibility of the 
>>>>>>>> calling
>>>>>>>> code to clean up if the higher level (multiple statements) 
>>>>>>>> operation
>>>>>>>> that A is embedded in, fails.
>>>>>>>>
>>>>>>>> And that's what Marginean's original C++ ScopeGuard was designed
>>>>>>>> for,
>>>>>>>> and what the corresponding Python Cleanup class is designed for.
>>>>>>>
>>>>>>> And try: finally:, for that matter.
>>>>>>
>>>>>> Not to mention "with".
>>>>>>
>>>>>> Some other poster made the same error recently in this thread; it 
>>>>>> is a
>>>>>> common fallacy in discussions about programming, to assume that since
>>>>>> the same can be expressed using lower level constructs, those are all
>>>>>> that are required.
>>>>>>
>>>>>> If adopted as true it ultimately means the removal of all control
>>>>>> structures above the level of "if" and "goto" (except Python doesn't
>>>>>> have "goto").
>>>>>
>>>>> What I'm trying to explain is that the with: statement has a use even
>>>>> if Cleanup doesn't. Arguing that Cleanup doesn't improve on try:
>>>>> finally: does not mean that the with: statement doesn't improve on
>>>>> try: finally:.
>>>>
>>>> That's a different argument, essentially that you see no advantage for
>>>> your current coding patterns.
>>>>
>>>> It's unconnected to the argument I responded to.
>>>>
>>>> The argument that I responded to, that the possibility of expressing
>>>> things at the level of try:finally: means that a higher level construct
>>>> is superfluous, is still meaningless.
>>>
>>> I am attacking your premise that the "with Cleanup():" construct is
>>> higher level than try: finally:. It isn't. It provides the same level
>>> of abstraction as try: finally:.
>>>
>>> This is distinct from the accepted uses of the with: statement which
>>> *are* higher level than try: finally: and which do confer practical
>>> benefits over using try: finally: despite being syntactical sugar for
>>> try: finally:.
>>>
>>>>>>>>> Both formulations can be correct (and both work perfectly fine 
>>>>>>>>> with
>>>>>>>>> the chdir() example being used). Sometimes one is better than the
>>>>>>>>> other, and sometimes not. You can achieve both ways with either
>>>>>>>>> your
>>>>>>>>> Cleanup class or with try: finally:.
>>>>>>>>>
>>>>>>>>> I am still of the opinion that Cleanup is not an improvement over
>>>>>>>>> try:
>>>>>>>>> finally: and has the significant ugliness of forcing cleanup code
>>>>>>>>> into
>>>>>>>>> callables. This significantly limits what you can do in your
>>>>>>>>> cleanup
>>>>>>>>> code.
>>>>>>>>
>>>>>>>> Uhm, not really. :-) As I see it.
>>>>>>>
>>>>>>> Well, not being able to affect the namespace is a significant
>>>>>>> limitation. Sometimes you need to delete objects from the
>>>>>>> namespace in
>>>>>>> order to ensure that their refcounts go to zero and their cleanup
>>>>>>> code
>>>>>>> gets executed.
>>>>>>
>>>>>> Just a nit (I agree that a lambda can't do this, but as to what's
>>>>>> required): assigning None is sufficient for that[1].
>>>>>
>>>>> Yes, but no callable is going to allow you to assign None to names in
>>>>> that namespace, either. Not without sys._getframe() hackery, in any
>>>>> case.
>>>>>
>>>>>> However, note that the current language doesn't guarantee such
>>>>>> cleanup,
>>>>>> at least as far as I know.
>>>>>>
>>>>>> So while it's good practice to support it, to do everything to let it
>>>>>> happen, it's presumably bad practice to rely on it happening.
>>>>>>
>>>>>>
>>>>>>> Tracebacks will keep the namespace alive and all objects in it.
>>>>>>
>>>>>> Thanks!, I hadn't thought of connecting that to general cleanup
>>>>>> actions.
>>>>>>
>>>>>> It limits the use of general "with" in the same way.
>>>>>
>>>>> Not really.
>>>>
>>>> Sorry, it limits general 'with' in /exactly/ the same way.
>>>>
>>>>> It's easy to write context managers that do that [delete objects from
>>>>> the namespace].
>>>>
>>>> Sorry, no can do, as far as I know; your following example quoted below
>>>> is an example of /something else/.
>>>
>>> Okay, so what do you mean by 'the use of general "with"'? I'm talking
>>> about writing a context manager or using the @contextmanager decorator
>>> to do some initialization and then later cleaning up that
>>> initialization. That cleaning up may entail deleting an object. You
>>> are correct that the context manager can't affect the namespace of the
>>> with: clause, but that's not the initialization that it would need to
>>> clean up.
>>>
>>> Yes, you can write code with a with: statement where you try to clean
>>> up stuff that happened inside of the clause (you did), but that's not
>>> how the with: statement was ever intended to be used nor is it good
>>> practice to do so because of that limitation. Context managers are
>>> designed to initialize specific things, then clean them up. I thought
>>> you were talking about the uses of the with: statement as described in
>>> PEP-343, not every possible misuse of the with: statement.
>>
>> I'm not the one talking about removing variables or that "it's easy to
>> write context managers that do that".
>>
>> You are the one talking about that.
>>
>> So I have really not much to add.
>>
>> It seems that you're now agreeing with me that former is not good
>> practice and that the latter is impossible to do portably, but you now
>> argue against your earlier stand as if that was something that I had put
>> forward.
> 
> No, I'm still saying that sometimes you do need to remove variables that 
> you initialized in the initialization section. Writing a purpose-built 
> context manager which encapsulates the initialization and finalization 
> allows you to do this easily. Putting the initialization section inside 
> the with: clause, as you do, and requiring cleanup code to be put into 
> callables makes this hard.
> 
>> It's a bit confusing when you argue against your own statements.
>>
>>>> And adding on top of irrelevancy, for the pure technical aspect it can
>>>> be accomplished in the same way using Cleanup (I provide an example
>>>> below).
>>>>
>>>> However, doing that would generally be worse than pointless since with
>>>> good coding practices the objects would become unreferenced anyway.
>>>>
>>>>
>>>>> You put the initialization code in the __enter__() method, assign
>>>>> whatever objects you want to keep around through the with: clause as
>>>>> attributes on the manager, then delete those attributes in the
>>>>> __exit__().
>>>>
>>>> Analogously, if one were to do this thing, then it could be 
>>>> accomplished
>>>> using a Cleanup context manager as follows:
>>>>
>>>> foo = lambda: None
>>>> foo.x = create_some_object()
>>>> at_cleanup.call( lambda o = foo: delattr( o, "x" ) )
>>>>
>>>> ... except that
>>>>
>>>> 1) for a once-only case this is less code :-)
>>>
>>> Not compared to a try: finally:, it isn't.
>>
>> Again, this context shifting is bewildering. As you can see, quoted
>> above, you were talking about a situation where you would have defined a
>> context manager, presumably because a 'try' would not in your opinion be
>> simpler for whatever it was that you had in mind. But you are responding
>> to the code I offered as if it was an alternative to something where you
>> would find a 'try' to be simplest.
> 
> I have consistently put forward that for once-only cases, try: finally: 
> is a preferable construct to "with Cleanup():". I also put forward that 
> for repetitive cases, a purpose-built context manager is preferable. I 
> note that for both try: finally: and a purpose-built context manager, 
> modifying the namespace is easy to do while it is difficult and less 
> readable to do with "with Cleanup():". Since you were claiming that the 
> "generic with:" would have the same problem as "with Cleanup():" I was 
> trying to explain for why all of the intended use cases of the with: 
> statement (the purpose-built context managers), there is no problem. 
> Capisce?
> 
>>>> 2) it is a usage that I wouldn't recommend; instead I recommend 
>>>> adopting
>>>> good
>>>> coding practices where object references aren't kept around.
>>>
>>> Many of the use cases of the with: statement involve creating an
>>> object (like a lock or a transaction object), keeping it around for
>>> the duration of the "# Do stuff" block, and then finalizing it.
>>>
>>>>> Or, you use the @contextmanager decorator to turn a generator into a
>>>>> context manager, and you just assign to local variables and del them
>>>>> in the finally: clause.
>>>>
>>>> Uhm, you don't need a 'finally' clause when you define a context
>>>> manager.
>>>
>>> When you use the @contextmanager decorator, you almost always do. See
>>> the Examples section of PEP 343:
>>>
>>> http://www.python.org/dev/peps/pep-0343/
>>>
>>>> Additionally, you don't need to 'del' the local variables in
>>>> @contextmanager decorated generator.
>>>>
>>>> The local variables cease to exist automatically.
>>>
>>> True.
>>>
>>>>> What you can't do is write a generic context manager where the
>>>>> initialization happens inside the with: clause and the cleanup actions
>>>>> are registered callables. That does not allow you to affect the
>>>>> namespace.
>>>>
>>>> If you mean that you can't introduce direct local variables and have
>>>> them deleted by "registered callables" in a portable way, then right.
>>>>
>>>> But I can't think of any example where that would be relevant; in
>>>> particular what matters for supporting on-destruction cleanup is 
>>>> whether
>>>> you keep any references or not, not whether you have a local 
>>>> variable of
>>>> any given name.
>>>
>>> Well, local variables keep references to objects. Variable assignment
>>> followed by deletion is a very readable way to keep an object around
>>> for a while then remove it later. If you have to go through less
>>> readable contortions to keep the object around when it needs to be and
>>> clean it up later, then that is a mark against your approach.
>>
>> Sorry, as with the places noted above, I can't understand what you're
>> trying to say here. I don't recommend coding practices where you keep
>> object references around,
> 
> Then how do you clean up locks and transaction objects and similar 
> things if you don't keep them around during the code suite? Creating an 
> object, keeping it around while executing a specific chunk of code, and 
> finalizing it safely is a prime use case for these kinds of constructs.

Where you need a scope, create a scope.

That is, put the logic in a routine.

However, most objects aren't of the sort the requiring to become unreferenced. 
Those that require cleanup can be cleaned up and left as zombies, like calling 
'close' on a file. And so in the general case what you're discussing, how to get 
rid of object references, is a non-issue  --  irrelevant.


>> and you have twice quoted that above. I don't
>> have any clue what "contortions" you are talking about, it must be
>> something that you imagine.
> 
> These are the contortions:
> 
>   foo = lambda: None
>   foo.x = create_some_object()
>   at_cleanup.call( lambda o = foo: delattr( o, "x" ) )

That's code that demonstrates something very different, in response to your 
description of doing this.

Since I half suspected that it could be taken out of context I followed that 
immediately with

<quote>
   2) it is a usage that I wouldn't recommend; instead I recommend adopting
      good coding practices where object references aren't kept around.
</quote>

I'm sorry but I don't know how to make that more clear.


Cheers & hth.,

- Alf