Possibly Pythonic Tail Call Optimization (TCO/TRE)

[Chris Angelico]

On Fri, Jul 17, 2015 at 10:54 PM, Antoon Pardon
<antoon.pardon at rece.vub.ac.be> wrote:
> On 07/17/2015 01:49 PM, Chris Angelico wrote:
>> On Fri, Jul 17, 2015 at 9:43 PM, Antoon Pardon
>> <antoon.pardon at rece.vub.ac.be> wrote:
>>
>>
>>> Sure, but in this case, the generator is still active. The Runtime
>>> would be able to jump to and somehow activates it's stack record
>>> for the next value. So why would we expect it to be impossible to
>>> include this trace back record in a stack trace?
>> Python could also give you stack traces for any other threads that are
>> concurrently running, on the off-chance that one of them affected it.
>> But the only influence the generator has on the loop is to yield a
>> value or signal termination; if an exception is thrown in the loop
>> itself, the local name 'stuff' should have all the information about
>> that cause.
>
> But the local name 'stuff' may only have the information for the immediate
> cause. The underlying cause may be available in the generator. Suppose
> you have a generator that should only generate positive numbers that you
> use to divide some other number by. Your loop crashes because of a DivideByZeroError
> Sure the local name shows the dividor to be zero, but you have no
> information on why your generator produced a zero, but there may be a
> clue in the trace back record of the generator.

Indeed, but there's nothing special about generators here. The same
sequence could have been a concrete list, or it could have been some
other kind of iterator (any object with __iter__ and __next__), which
won't have a stack frame. Special cases aren't special enough to warp
exception handling around.

>>  Python isn't a mind-reader, no matter how much it may look
>> like one, and it can't know that this function's return value should
>> be shown as part of a completely different function's stack trace.
>
> It is not a matter of mindreading. And it is not a completely different
> functions stack trace. It is the trace back record of a generator that
> is used by the process/thread that crashed. And AFAIK an active generator
> belongs to one specific thread. You can't have it yield a value to a different
> thread and you can't send it a value from an other thread. So I really
> see no reason to exclude the trace back records of active generators
> from a stack trace of a crashed thread.

No, generators are fine across threads:

rosuav at sikorsky:~$ python3 threadgen.py
Starting!
First yielded value
Continuing!
Second yielded value
Terminating.
Traceback (most recent call last):
  File "threadgen.py", line 20, in <module>
    print(next(gen))
StopIteration
rosuav at sikorsky:~$ cat threadgen.py
import threading
import time

def thread():
    time.sleep(0.5)
    print(next(gen))

threading.Thread(target=thread).start()

def generator():rosuav at sikorsky:~$

    print("Starting!")
    yield "First yielded value"
    print("Continuing!")
    yield "Second yielded value"
    print("Terminating.")

gen = generator()
print(next(gen))
time.sleep(1)
print(next(gen))
rosuav at sikorsky:~$

In fact, a generator doesn't have a stack unless it's currently
executing, so all you could get is whatever's actually inside it (that
is, if there's a deep tree of 'yield from's, you could dig up that
part of the stack). I'm not sure this would really help you very much.

ChrisA