[Python-Dev] Status of thread cancellation

[glyph at divmod.com]

On 04:24 pm, martin at v.loewis.de wrote:
>Jean-Paul Calderone schrieb:
>>>I inferred from Martin's proposal that he
>>>expected the thread to be able to catch the exception.  Perhaps he 
>>>can
>>>elaborate on what cleanup actions the dying thread will be allowed to
>>>perform.
>>
>>Perhaps he can.  Hopefully, he can specifically address these points:
>>
>>    1. A thread can throw a ThreadDeath exception almost anywhere. All
>>       synchronized methods and blocks would have to be studied in 
>>great
>>       detail, with this in mind.
>>
>>    2. A thread can throw a second ThreadDeath exception while cleaning 
>>up
>>       from the first (in the catch or finally clause). Cleanup would 
>>have
>>       to repeated till it succeeded. The code to ensure this would be 
>>quite
>>       complex.
>
>Clearly, a thread need to have its finally blocks performed in response
>to a cancellation request. These issues are real, however, they apply
>to any asynchronous exception, not just to thread cancellation.

To be sure, the problem does apply to all asynchronous exceptions. 
That's why it is generally understood that a program which has received 
an asynchronous exception cannot continue.
>In Python, we already have an asynchronous exception: 
>KeyboardInterrupt.
>This suffers from the same problems: a KeyboadInterrupt also can occur
>at any point, interrupting code in the middle of its finally-blocks.
>The other exception that is nearly-asynchronous is OutOfMemoryError,
>which can occur at nearly any point (but of course, never occurs in
>practice).

KeyboardInterrupt and MemoryError share a common feature which forced 
thread termination does not: nobody reasonably expects the program to 
keep running after they have been raised.  Indeed, programs are written 
with the expectation that MemoryError will never occur, and if it does, 
the user is not surprised to find them in an inconsistent state.  In any 
situation where a MemoryError may reasonably be expected - that is to 
say, a specific, large allocation of a single block of memory - it can 
be trapped as if it were not asynchronous.

Long-running Python programs which expect to need to do serious clean-up 
in the face of interrupts, in fact, block KeyboardInterrupt by 
registering their own interrupt handlers (Zope, Twisted).

Developers who think they want thread cancellation inevitably believe 
they can, if they are "sufficiently careful", implement operating- 
system-like scheduling features by starting arbitrary user code and then 
providing "terminate", "pause", and "resume" commands.  That was the 
original intent of these (now removed) Java APIs, and that is why they 
were removed: you can't do this.  It's impossible.

Asynchronous exceptions are better than immediate termination because 
they allow for code which is allocating scarce or fragile resources to 
have a probabilistically better chance of cleaning up.  However, nobody 
writes code like this:

def addSomeStuff(self, volume, mass):
    self.volume += volume
    try:
        self.mass += mass
    except AsynchronousInterrupt:
        while 1:
            try:
                self.volume -= volume
                break
            except AsynchronousInterrupt:
                pass

and nobody is going to start if the language provides thread 
termination.  Async-Exception-Safe Python code is, and will be, as rare 
as POSIX Async-Safe functions, which means at best you will be able to 
call a thread cancellation API and have it _appear_ to work in some 
circumstances.  In any system which uses Python code not explicitly 
designed to support asynchronous exceptions (let's say, the standard 
library) it will be completely impossible to write correct code.

I'm not a big fan of shared-state-threading, but it does allow for a 
particular programming model.  Threading provides you some guarantees. 
You can't poke around on the heap, but you know that your stack, and 
your program counter, are inviolate.  You can reason about, if not quite 
test, the impact of sharing a piece of state on the heap; its 
destructive methods need to be synchronized along with the read methods 
that interact with it.

Asynchronous exceptions destroy all hope of sanity.  Your program might 
suddenly perform a nonlocal exit _anywhere_ except, maybe, inside a 
"finally" block.   This literally encourages some people that program in 
environments where asynchronous exceptions are possible (.NET, in 
particular) to put huge chunks of application code inside finally 
blocks.  They generally look like this:

    try {}
    finally {
        // entire application here
    }

because that is really the only way you can hope to write code that will 
function robustly in such an environment.
>So yes, it would be good if Python's exception handling supported
>asynchronous exceptions in a sensible way. I have to research somewhat
>more, but I think the standard solution to the problem in operating
>system (i.e. disabling interrupts at certain points, explicitly
>due to code or implicitly as a result of entering the interrupt
>handler) may apply.

For one thing, the Python core code is not operating system kernel code 
and it is unlikely that the techniques found there will apply. 
Interrupts, in particular, are nothing at all like exceptions, and have 
a completely different impact on running code (which, since it is kernel 
code, is written much more carefully and under an entirely different set 
of constraints than Python application, or even framework, code).

Can you suggest any use-cases for thread termination which will *not* 
result in a completely broken and unpredictable heap after the thread 
has died?  If you can think of such a case, are you sure it wouldn't be 
better served by a set of threads communicating over queues and sending 
'Stop' objects to each other to indicate termination at a point in queue 
rather than a forced termination via exception?

Just in case it's not clear from the other things I've said: this is a 
terrible, terrible idea, and I am shocked that it is even being 
*considered* for inclusion in Python.  As a foolish youth, I wasted many 
months trying to get a program that used Java's (then not deprecated) 
asynchronous exception APIs to behave properly.  It wasn't possible 
then, and it isn't possible now.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070315/6a356b17/attachment-0001.html