From chris.jerdonek at gmail.com Thu Nov 2 19:51:10 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Thu, 2 Nov 2017 16:51:10 -0700 Subject: [Async-sig] coroutine function vs. function returning awaitable Message-ID: Hi, I have a 2-part question: 1) Say you have a function that returns an awaitable (but not a coroutine object). Are there any advantages or disadvantages to replacing this with a function for which asyncio.iscoroutinefunction() returns True and asyncio.iscoroutine() returns True for the return value? An advantage might be, say, additional checks that asyncio performs. 2) If there are advantages, is there a straightforward way to construct such a replacement? In the case I have in mind (from an open source project), the "function" you'd be starting with is the class constructor for an async context manager that is also awaitable. So the class has both __aenter__ and __await__ methods. As an example naive attempt, the following _wouldn't_ work: @asyncio.coroutine def connect(): return AsyncConnectManager() The reason this doesn't work is that wrapping connect() in @asyncio.coroutine() means the return value is no longer an async context manager (it no longer has an __aenter__ method). You have to await to get the underlying async context manager. Thanks, --Chris From guido at python.org Thu Nov 2 22:14:22 2017 From: guido at python.org (Guido van Rossum) Date: Thu, 2 Nov 2017 19:14:22 -0700 Subject: [Async-sig] coroutine function vs. function returning awaitable In-Reply-To: References: Message-ID: Re (1), there are only two places where asyncio calls `iscouroutinefunction()`, and in both cases it raises a TypeError to remind the user that a coroutine/awaitable object is required. So, it doesn't really offer much of an advantage and I wouldn't worry about it. On Thu, Nov 2, 2017 at 4:51 PM, Chris Jerdonek wrote: > Hi, > > I have a 2-part question: > > 1) Say you have a function that returns an awaitable (but not a > coroutine object). Are there any advantages or disadvantages to > replacing this with a function for which asyncio.iscoroutinefunction() > returns True and asyncio.iscoroutine() returns True for the return > value? An advantage might be, say, additional checks that asyncio > performs. > > 2) If there are advantages, is there a straightforward way to > construct such a replacement? In the case I have in mind (from an open > source project), the "function" you'd be starting with is the class > constructor for an async context manager that is also awaitable. So > the class has both __aenter__ and __await__ methods. > > As an example naive attempt, the following _wouldn't_ work: > > @asyncio.coroutine > def connect(): > return AsyncConnectManager() > > The reason this doesn't work is that wrapping connect() in > @asyncio.coroutine() means the return value is no longer an async > context manager (it no longer has an __aenter__ method). You have to > await to get the underlying async context manager. > > Thanks, > --Chris > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Thu Nov 2 22:49:03 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Fri, 03 Nov 2017 02:49:03 +0000 Subject: [Async-sig] coroutine function vs. function returning awaitable In-Reply-To: References: Message-ID: Thanks. And how about making asyncio.iscoroutine() return True for the return value? It looks like asyncio calls that one in a half-dozen places or so. ?Chris On Thu, Nov 2, 2017 at 7:15 PM Guido van Rossum wrote: > Re (1), there are only two places where asyncio calls > `iscouroutinefunction()`, and in both cases it raises a TypeError to remind > the user that a coroutine/awaitable object is required. So, it doesn't > really offer much of an advantage and I wouldn't worry about it. > > On Thu, Nov 2, 2017 at 4:51 PM, Chris Jerdonek > wrote: > >> Hi, >> >> I have a 2-part question: >> >> 1) Say you have a function that returns an awaitable (but not a >> coroutine object). Are there any advantages or disadvantages to >> replacing this with a function for which asyncio.iscoroutinefunction() >> returns True and asyncio.iscoroutine() returns True for the return >> value? An advantage might be, say, additional checks that asyncio >> performs. >> >> 2) If there are advantages, is there a straightforward way to >> construct such a replacement? In the case I have in mind (from an open >> source project), the "function" you'd be starting with is the class >> constructor for an async context manager that is also awaitable. So >> the class has both __aenter__ and __await__ methods. >> >> As an example naive attempt, the following _wouldn't_ work: >> >> @asyncio.coroutine >> def connect(): >> return AsyncConnectManager() >> >> The reason this doesn't work is that wrapping connect() in >> @asyncio.coroutine() means the return value is no longer an async >> context manager (it no longer has an __aenter__ method). You have to >> await to get the underlying async context manager. >> >> Thanks, >> --Chris >> > _______________________________________________ >> Async-sig mailing list >> Async-sig at python.org >> https://mail.python.org/mailman/listinfo/async-sig >> Code of Conduct: https://www.python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Nov 2 23:16:31 2017 From: guido at python.org (Guido van Rossum) Date: Thu, 2 Nov 2017 20:16:31 -0700 Subject: [Async-sig] coroutine function vs. function returning awaitable In-Reply-To: References: Message-ID: iscoroutine() is usually used to turn something into a Future. Is that what you want? (TBH I'm not sure why you're asking all this.) On Thu, Nov 2, 2017 at 7:49 PM, Chris Jerdonek wrote: > Thanks. And how about making asyncio.iscoroutine() return True for the > return value? It looks like asyncio calls that one in a half-dozen places > or so. > > ?Chris > > On Thu, Nov 2, 2017 at 7:15 PM Guido van Rossum wrote: > >> Re (1), there are only two places where asyncio calls >> `iscouroutinefunction()`, and in both cases it raises a TypeError to remind >> the user that a coroutine/awaitable object is required. So, it doesn't >> really offer much of an advantage and I wouldn't worry about it. >> >> On Thu, Nov 2, 2017 at 4:51 PM, Chris Jerdonek >> wrote: >> >>> Hi, >>> >>> I have a 2-part question: >>> >>> 1) Say you have a function that returns an awaitable (but not a >>> coroutine object). Are there any advantages or disadvantages to >>> replacing this with a function for which asyncio.iscoroutinefunction() >>> returns True and asyncio.iscoroutine() returns True for the return >>> value? An advantage might be, say, additional checks that asyncio >>> performs. >>> >>> 2) If there are advantages, is there a straightforward way to >>> construct such a replacement? In the case I have in mind (from an open >>> source project), the "function" you'd be starting with is the class >>> constructor for an async context manager that is also awaitable. So >>> the class has both __aenter__ and __await__ methods. >>> >>> As an example naive attempt, the following _wouldn't_ work: >>> >>> @asyncio.coroutine >>> def connect(): >>> return AsyncConnectManager() >>> >>> The reason this doesn't work is that wrapping connect() in >>> @asyncio.coroutine() means the return value is no longer an async >>> context manager (it no longer has an __aenter__ method). You have to >>> await to get the underlying async context manager. >>> >>> Thanks, >>> --Chris >>> >> _______________________________________________ >>> Async-sig mailing list >>> Async-sig at python.org >>> https://mail.python.org/mailman/listinfo/async-sig >>> Code of Conduct: https://www.python.org/psf/codeofconduct/ >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Thu Nov 2 23:33:51 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Thu, 2 Nov 2017 20:33:51 -0700 Subject: [Async-sig] coroutine function vs. function returning awaitable In-Reply-To: References: Message-ID: I'm asking because a project I've been contributing to has this "bug" reported: https://github.com/aaugustin/websockets/issues/300 So I'm trying to see if it needs to be fixed or if it can be left alone. More generally, this made me wonder if there are any advantages to ensuring that asyncio.iscoroutine() returns True for an object if the object is meant to be used like a coroutine object. (For example, perhaps this is what will give you the nice "coroutine object hasn't been awaited" warnings.) It looks like types.coroutine() might be the answer if there are advantages (but I haven't tried it): https://www.python.org/dev/peps/pep-0492/#types-coroutine --Chris On Thu, Nov 2, 2017 at 8:16 PM, Guido van Rossum wrote: > iscoroutine() is usually used to turn something into a Future. Is that what > you want? (TBH I'm not sure why you're asking all this.) > > On Thu, Nov 2, 2017 at 7:49 PM, Chris Jerdonek > wrote: >> >> Thanks. And how about making asyncio.iscoroutine() return True for the >> return value? It looks like asyncio calls that one in a half-dozen places or >> so. >> >> ?Chris >> >> On Thu, Nov 2, 2017 at 7:15 PM Guido van Rossum wrote: >>> >>> Re (1), there are only two places where asyncio calls >>> `iscouroutinefunction()`, and in both cases it raises a TypeError to remind >>> the user that a coroutine/awaitable object is required. So, it doesn't >>> really offer much of an advantage and I wouldn't worry about it. >>> >>> On Thu, Nov 2, 2017 at 4:51 PM, Chris Jerdonek >>> wrote: >>>> >>>> Hi, >>>> >>>> I have a 2-part question: >>>> >>>> 1) Say you have a function that returns an awaitable (but not a >>>> coroutine object). Are there any advantages or disadvantages to >>>> replacing this with a function for which asyncio.iscoroutinefunction() >>>> returns True and asyncio.iscoroutine() returns True for the return >>>> value? An advantage might be, say, additional checks that asyncio >>>> performs. >>>> >>>> 2) If there are advantages, is there a straightforward way to >>>> construct such a replacement? In the case I have in mind (from an open >>>> source project), the "function" you'd be starting with is the class >>>> constructor for an async context manager that is also awaitable. So >>>> the class has both __aenter__ and __await__ methods. >>>> >>>> As an example naive attempt, the following _wouldn't_ work: >>>> >>>> @asyncio.coroutine >>>> def connect(): >>>> return AsyncConnectManager() >>>> >>>> The reason this doesn't work is that wrapping connect() in >>>> @asyncio.coroutine() means the return value is no longer an async >>>> context manager (it no longer has an __aenter__ method). You have to >>>> await to get the underlying async context manager. >>>> >>>> Thanks, >>>> --Chris >>>> >>>> _______________________________________________ >>>> Async-sig mailing list >>>> Async-sig at python.org >>>> https://mail.python.org/mailman/listinfo/async-sig >>>> Code of Conduct: https://www.python.org/psf/codeofconduct/ >>> >>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) > > > > > -- > --Guido van Rossum (python.org/~guido) From guido at python.org Fri Nov 3 00:08:41 2017 From: guido at python.org (Guido van Rossum) Date: Thu, 2 Nov 2017 21:08:41 -0700 Subject: [Async-sig] coroutine function vs. function returning awaitable In-Reply-To: References: Message-ID: It looks like what you have in that library is something that's awaitable but not a coroutine. That's totally fine. (Other examples exist, e.g. Futures!) On Thu, Nov 2, 2017 at 8:33 PM, Chris Jerdonek wrote: > I'm asking because a project I've been contributing to has this "bug" > reported: > > https://github.com/aaugustin/websockets/issues/300 > > So I'm trying to see if it needs to be fixed or if it can be left alone. > > More generally, this made me wonder if there are any advantages to > ensuring that asyncio.iscoroutine() returns True for an object if the > object is meant to be used like a coroutine object. (For example, > perhaps this is what will give you the nice "coroutine object hasn't > been awaited" warnings.) > > It looks like types.coroutine() might be the answer if there are > advantages (but I haven't tried it): > https://www.python.org/dev/peps/pep-0492/#types-coroutine > > --Chris > > On Thu, Nov 2, 2017 at 8:16 PM, Guido van Rossum wrote: > > iscoroutine() is usually used to turn something into a Future. Is that > what > > you want? (TBH I'm not sure why you're asking all this.) > > > > On Thu, Nov 2, 2017 at 7:49 PM, Chris Jerdonek > > > wrote: > >> > >> Thanks. And how about making asyncio.iscoroutine() return True for the > >> return value? It looks like asyncio calls that one in a half-dozen > places or > >> so. > >> > >> ?Chris > >> > >> On Thu, Nov 2, 2017 at 7:15 PM Guido van Rossum > wrote: > >>> > >>> Re (1), there are only two places where asyncio calls > >>> `iscouroutinefunction()`, and in both cases it raises a TypeError to > remind > >>> the user that a coroutine/awaitable object is required. So, it doesn't > >>> really offer much of an advantage and I wouldn't worry about it. > >>> > >>> On Thu, Nov 2, 2017 at 4:51 PM, Chris Jerdonek < > chris.jerdonek at gmail.com> > >>> wrote: > >>>> > >>>> Hi, > >>>> > >>>> I have a 2-part question: > >>>> > >>>> 1) Say you have a function that returns an awaitable (but not a > >>>> coroutine object). Are there any advantages or disadvantages to > >>>> replacing this with a function for which asyncio.iscoroutinefunction() > >>>> returns True and asyncio.iscoroutine() returns True for the return > >>>> value? An advantage might be, say, additional checks that asyncio > >>>> performs. > >>>> > >>>> 2) If there are advantages, is there a straightforward way to > >>>> construct such a replacement? In the case I have in mind (from an open > >>>> source project), the "function" you'd be starting with is the class > >>>> constructor for an async context manager that is also awaitable. So > >>>> the class has both __aenter__ and __await__ methods. > >>>> > >>>> As an example naive attempt, the following _wouldn't_ work: > >>>> > >>>> @asyncio.coroutine > >>>> def connect(): > >>>> return AsyncConnectManager() > >>>> > >>>> The reason this doesn't work is that wrapping connect() in > >>>> @asyncio.coroutine() means the return value is no longer an async > >>>> context manager (it no longer has an __aenter__ method). You have to > >>>> await to get the underlying async context manager. > >>>> > >>>> Thanks, > >>>> --Chris > >>>> > >>>> _______________________________________________ > >>>> Async-sig mailing list > >>>> Async-sig at python.org > >>>> https://mail.python.org/mailman/listinfo/async-sig > >>>> Code of Conduct: https://www.python.org/psf/codeofconduct/ > >>> > >>> > >>> > >>> > >>> -- > >>> --Guido van Rossum (python.org/~guido) > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Sat Nov 11 00:52:14 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Fri, 10 Nov 2017 21:52:14 -0800 Subject: [Async-sig] awaiting task is not chaining exception Message-ID: Hi, I recently encountered a situation with asyncio where the stack trace is getting truncated: an exception isn't getting chained as expected. I was able to reduce it down to the code below. The reduced case seems like a pattern that can come up a lot, and I wasn't able to find an issue on the CPython tracker, so I'm wondering if I'm doing something wrong or if the behavior is deliberate. The way I would describe it is that (at least in this case) awaiting on a task that raises an exception doesn't set __context__ when awaiting on the task inside an "except" block. Here is the code: import asyncio async def raise_error(): raise RuntimeError('exception #2') async def main(): try: raise RuntimeError('exception #1') except Exception: future = raise_error() # Uncommenting the next line makes exc.__context__ None. # future = asyncio.ensure_future(future) try: await future except Exception as exc: print(f'exc: {exc!r}, context: {exc.__context__!r}') raise asyncio.get_event_loop().run_until_complete(main()) Here is the output of the above, which works as expected: exc: RuntimeError('exception #2',), context: RuntimeError('exception #1',) Traceback (most recent call last): File "test.py", line 8, in main raise RuntimeError('exception #1') RuntimeError: exception #1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "test.py", line 20, in asyncio.get_event_loop().run_until_complete(main()) File "/.../python3.6/asyncio/base_events.py", line 466, in run_until_complete return future.result() File "test.py", line 15, in main await future File "test.py", line 4, in raise_error raise RuntimeError('exception #2') RuntimeError: exception #2 And here is the unexpected output when the commented line is uncommented (exception not getting chained): exc: RuntimeError('exception #2',), context: None Traceback (most recent call last): File "test.py", line 22, in asyncio.get_event_loop().run_until_complete(main()) File "/.../python3.6/asyncio/base_events.py", line 466, in run_until_complete return future.result() File "test.py", line 17, in main await future File "test.py", line 6, in raise_error raise RuntimeError('exception #2') RuntimeError: exception #2 --Chris From njs at pobox.com Sat Nov 11 03:39:24 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 11 Nov 2017 00:39:24 -0800 Subject: [Async-sig] awaiting task is not chaining exception In-Reply-To: References: Message-ID: On Fri, Nov 10, 2017 at 9:52 PM, Chris Jerdonek wrote: > Hi, I recently encountered a situation with asyncio where the stack > trace is getting truncated: an exception isn't getting chained as > expected. > > I was able to reduce it down to the code below. > > The reduced case seems like a pattern that can come up a lot, and I > wasn't able to find an issue on the CPython tracker, so I'm wondering > if I'm doing something wrong or if the behavior is deliberate. I think what you're seeing is collateral damage from some known bugginess in the generator/coroutine .throw() method: https://bugs.python.org/issue29587 In trio I work around this by never using throw(); instead I send() in the exception and re-raise it inside the coroutine: https://github.com/python-trio/trio/blob/389f1e1e01b410756e2833cffb992fd1ff856ae5/trio/_core/_run.py#L1491-L1498 But asyncio doesn't do this -- when an asyncio.Task awaits an asyncio.Future and the Future raises, the exception is throw()n into the Task, triggering the bug: https://github.com/python/cpython/blob/e184cfd7bf8bcfd160e3b611d4351ca3ce52d9e2/Lib/asyncio/tasks.py#L178 (If you try profiling your code you may also see weird/impossible results in cases like this, because throw() also messes up stack introspection.) -n -- Nathaniel J. Smith -- https://vorpus.org From chris.jerdonek at gmail.com Sat Nov 11 04:47:21 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 11 Nov 2017 01:47:21 -0800 Subject: [Async-sig] awaiting task is not chaining exception In-Reply-To: References: Message-ID: On Sat, Nov 11, 2017 at 12:39 AM, Nathaniel Smith wrote: > On Fri, Nov 10, 2017 at 9:52 PM, Chris Jerdonek > wrote: >> Hi, I recently encountered a situation with asyncio where the stack >> trace is getting truncated: an exception isn't getting chained as >> expected. >> >> I was able to reduce it down to the code below. >> >> The reduced case seems like a pattern that can come up a lot, and I >> wasn't able to find an issue on the CPython tracker, so I'm wondering >> if I'm doing something wrong or if the behavior is deliberate. > > I think what you're seeing is collateral damage from some known > bugginess in the generator/coroutine .throw() method: > https://bugs.python.org/issue29587 Ah, thanks for the great explanation, Nathaniel! >From the original bug report Nathaniel filed above: > It's likely that more people will run into this in the future as async/await becomes more widely used. ... :) --Chris From chris.jerdonek at gmail.com Sun Nov 12 05:53:19 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sun, 12 Nov 2017 02:53:19 -0800 Subject: [Async-sig] awaiting task is not chaining exception In-Reply-To: References: Message-ID: By the way, since we're already on the subject of asyncio tasks and (truncated) stack traces, this looks like a good opportunity to ask a question that's been on my mind for a while: There's a mysterious note at the end of the documentation of asyncio.Task's get_stack() method, where it says-- > For reasons beyond our control, only one stack frame is returned for a suspended coroutine. (https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.get_stack ) What does the "For reasons beyond our control" mean? What is it that can possibly be beyond the control of Python? --Chris On Sat, Nov 11, 2017 at 1:47 AM, Chris Jerdonek wrote: > On Sat, Nov 11, 2017 at 12:39 AM, Nathaniel Smith wrote: >> On Fri, Nov 10, 2017 at 9:52 PM, Chris Jerdonek >> wrote: >>> Hi, I recently encountered a situation with asyncio where the stack >>> trace is getting truncated: an exception isn't getting chained as >>> expected. >>> >>> I was able to reduce it down to the code below. >>> >>> The reduced case seems like a pattern that can come up a lot, and I >>> wasn't able to find an issue on the CPython tracker, so I'm wondering >>> if I'm doing something wrong or if the behavior is deliberate. >> >> I think what you're seeing is collateral damage from some known >> bugginess in the generator/coroutine .throw() method: >> https://bugs.python.org/issue29587 > > Ah, thanks for the great explanation, Nathaniel! > > From the original bug report Nathaniel filed above: >> It's likely that more people will run into this in the future as async/await becomes more widely used. ... > > :) > > --Chris From guido at python.org Sun Nov 12 10:27:31 2017 From: guido at python.org (Guido van Rossum) Date: Sun, 12 Nov 2017 07:27:31 -0800 Subject: [Async-sig] awaiting task is not chaining exception In-Reply-To: References: Message-ID: On Sun, Nov 12, 2017 at 2:53 AM, Chris Jerdonek wrote: > By the way, since we're already on the subject of asyncio tasks and > (truncated) stack traces, this looks like a good opportunity to ask a > question that's been on my mind for a while: > > There's a mysterious note at the end of the documentation of > asyncio.Task's get_stack() method, where it says-- > > > For reasons beyond our control, only one stack frame is returned for a > suspended coroutine. > (https://docs.python.org/3/library/asyncio-task.html# > asyncio.Task.get_stack ) > > What does the "For reasons beyond our control" mean? What is it that > can possibly be beyond the control of Python? > It's an odd phrasing, but it refers to the fact that a suspended generator frame (which is what a coroutine really is) does not have a "back link" to the frame that called it. Whenever a generator yields (or a coroutine awaits) its frame is disconnected from the current call stack, control is passed to the top frame left on that stack, and the single generator frame is just held on to by the generator object. When you call next() on that, it will be pushed on top of whatever is the current stack (i.e. whatever calls next()), which *may* be a completely different stack configuration than when it was suspended. That's all. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Nov 12 23:28:47 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 12 Nov 2017 20:28:47 -0800 Subject: [Async-sig] awaiting task is not chaining exception In-Reply-To: References: Message-ID: On Sun, Nov 12, 2017 at 7:27 AM, Guido van Rossum wrote: > On Sun, Nov 12, 2017 at 2:53 AM, Chris Jerdonek > wrote: >> >> By the way, since we're already on the subject of asyncio tasks and >> (truncated) stack traces, this looks like a good opportunity to ask a >> question that's been on my mind for a while: >> >> There's a mysterious note at the end of the documentation of >> asyncio.Task's get_stack() method, where it says-- >> >> > For reasons beyond our control, only one stack frame is returned for a >> > suspended coroutine. >> >> (https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.get_stack >> ) >> >> What does the "For reasons beyond our control" mean? What is it that >> can possibly be beyond the control of Python? > > > It's an odd phrasing, but it refers to the fact that a suspended generator > frame (which is what a coroutine really is) does not have a "back link" to > the frame that called it. Whenever a generator yields (or a coroutine > awaits) its frame is disconnected from the current call stack, control is > passed to the top frame left on that stack, and the single generator frame > is just held on to by the generator object. When you call next() on that, it > will be pushed on top of whatever is the current stack (i.e. whatever calls > next()), which *may* be a completely different stack configuration than when > it was suspended. It is possible to get the await/yield from stack though, even when a generator/coroutine is suspended, using the gi_yieldfrom / cr_await attributes. Teaching Task.get_stack to do this would be a nice little enhancement. -n -- Nathaniel J. Smith -- https://vorpus.org From mehaase at gmail.com Tue Nov 14 09:54:31 2017 From: mehaase at gmail.com (Mark E. Haase) Date: Tue, 14 Nov 2017 09:54:31 -0500 Subject: [Async-sig] Simplifying stack traces for tasks? Message-ID: If an exception is thrown while the `asyncio` event loop is running, the stack trace is pretty complicated. Here's an example: Traceback (most recent call last): File "sample_client.py", line 616, in loop.run_until_complete(main_task) File "/usr/lib/python3.6/asyncio/base_events.py", line 437, in run_until_complete self.run_forever() File "/usr/lib/python3.6/asyncio/base_events.py", line 405, in run_forever self._run_once() File "/usr/lib/python3.6/asyncio/base_events.py", line 1382, in _run_once handle._run() File "/usr/lib/python3.6/asyncio/events.py", line 126, in _run self._callback(*self._args) File "/usr/lib/python3.6/asyncio/tasks.py", line 315, in _wakeup self._step() File "/usr/lib/python3.6/asyncio/tasks.py", line 239, in _step result = coro.send(None) File "sample_client.py", line 319, in main await actions[args.action](args, socket) File "/usr/lib/python3.6/asyncio/coroutines.py", line 105, in __next__ return self.gen.send(None) File "sample_client.py", line 558, in sync_job await asyncio.sleep(args.delay) File "/usr/lib/python3.6/asyncio/coroutines.py", line 105, in __next__ return self.gen.send(None) File "/usr/lib/python3.6/asyncio/tasks.py", line 522, in sleep future, result) File "/usr/lib/python3.6/asyncio/base_events.py", line 525, in call_later timer = self.call_at(self.time() + delay, callback, *args) File "/usr/lib/python3.6/asyncio/base_events.py", line 541, in call_at timer = events.TimerHandle(when, callback, args, self) File "/usr/lib/python3.6/asyncio/events.py", line 148, in __init__ super().__init__(callback, args, loop) File "/usr/lib/python3.6/asyncio/events.py", line 92, in __init__ self._source_traceback = traceback.extract_stack(sys._getframe(1)) File "/usr/lib/python3.6/traceback.py", line 207, in extract_stack stack = StackSummary.extract(walk_stack(f), limit=limit) File "/usr/lib/python3.6/traceback.py", line 352, in extract filename, lineno, name, lookup_line=False, locals=f_locals)) KeyboardInterrupt Most of the stack frames are inside asyncio (_run, _wakeup, _step) and reveal the mechanics of the event loop. The stack trace would be a lot easier to read (and more similar to a stack trace of an equivalent synchronous program) if it looked like this: Traceback (most recent call last): File "sample_client.py", line 616, in loop.run_until_complete(main_task) ... File "sample_client.py", line 319, in main await actions[args.action](args, socket) ... File "sample_client.py", line 558, in sync_job await asyncio.sleep(args.delay) KeyboardInterrupt I recognize that the event loop mechanics are probably useful in some circumstances, but in my experience working with asyncio for the last year, I've found the stack traces are generally too noisy. For a while, I was pasting stack traces into a text editor and cleaning them up by hand, but a few months ago I wrote a custom excepthook that generates stack traces similar to the one above. def async_excepthook(type_, exc, tb): cause_exc = None cause_str = None if exc.__cause__ is not None: cause_exc = exc.__cause__ cause_str = 'The above exception was the direct cause ' \ 'of the following exception:' elif exc.__context__ is not None and not exc.__suppress_context__: cause_exc = exc.__context__ cause_str = 'During handling of the above exception, ' \ 'another exception occurred:' if cause_exc: async_excepthook(type(cause_exc), cause_exc, cause_exc.__traceback__) if cause_str: print('\n{}\n'.format(cause_str)) print('Async Traceback (most recent call last):') for frame in traceback.extract_tb(tb): head, tail = os.path.split(frame.filename) if (head.endswith('asyncio') or tail == 'traceback.py') and \ frame.name.startswith('_'): print(' ...') continue print(' File "{}", line {}, in {}' .format(frame.filename, frame.lineno, frame.name)) print(' {}'.format(frame.line)) print('{}: {}'.format(type_.__name__, exc)) The meat of it is towards the bottom, "if head.endswith('asyncio')..."There are a lot of debatable details and this implementation is pretty hacky and clumsy, but I have found it valuable in my own usage, and I haven't yet missed the omitted stack frames. I'm posting here to get constructive criticism on the concept and would also like to hear if Curio or Trio have done anything like this. (Based on a quick skim of the docs, I don't think they are.) I might publish a PyPI package if anybody is interested. Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Tue Nov 14 10:08:30 2017 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Tue, 14 Nov 2017 15:08:30 +0000 Subject: [Async-sig] Simplifying stack traces for tasks? In-Reply-To: References: Message-ID: AFAIK Python never hides stdlib codelines in tracebacks. Why we should start to do it in asyncio? On Tue, Nov 14, 2017 at 4:54 PM Mark E. Haase wrote: > If an exception is thrown while the `asyncio` event loop is running, the > stack trace is pretty complicated. Here's an example: > > Traceback (most recent call last): > File "sample_client.py", line 616, in > loop.run_until_complete(main_task) > File "/usr/lib/python3.6/asyncio/base_events.py", line 437, in > run_until_complete > self.run_forever() > File "/usr/lib/python3.6/asyncio/base_events.py", line 405, in > run_forever > self._run_once() > File "/usr/lib/python3.6/asyncio/base_events.py", line 1382, in > _run_once > handle._run() > File "/usr/lib/python3.6/asyncio/events.py", line 126, in _run > self._callback(*self._args) > File "/usr/lib/python3.6/asyncio/tasks.py", line 315, in _wakeup > self._step() > File "/usr/lib/python3.6/asyncio/tasks.py", line 239, in _step > result = coro.send(None) > File "sample_client.py", line 319, in main > await actions[args.action](args, socket) > File "/usr/lib/python3.6/asyncio/coroutines.py", line 105, in > __next__ > return self.gen.send(None) > File "sample_client.py", line 558, in sync_job > await asyncio.sleep(args.delay) > File "/usr/lib/python3.6/asyncio/coroutines.py", line 105, in > __next__ > return self.gen.send(None) > File "/usr/lib/python3.6/asyncio/tasks.py", line 522, in sleep > future, result) > File "/usr/lib/python3.6/asyncio/base_events.py", line 525, in > call_later > timer = self.call_at(self.time() + delay, callback, *args) > File "/usr/lib/python3.6/asyncio/base_events.py", line 541, in > call_at > timer = events.TimerHandle(when, callback, args, self) > File "/usr/lib/python3.6/asyncio/events.py", line 148, in __init__ > super().__init__(callback, args, loop) > File "/usr/lib/python3.6/asyncio/events.py", line 92, in __init__ > self._source_traceback = traceback.extract_stack(sys._getframe(1)) > File "/usr/lib/python3.6/traceback.py", line 207, in extract_stack > stack = StackSummary.extract(walk_stack(f), limit=limit) > File "/usr/lib/python3.6/traceback.py", line 352, in extract > filename, lineno, name, lookup_line=False, locals=f_locals)) > KeyboardInterrupt > > Most of the stack frames are inside asyncio (_run, _wakeup, _step) and > reveal the mechanics of the event loop. The stack trace would be a lot > easier to read (and more similar to a stack trace of an equivalent > synchronous program) if it looked like this: > > Traceback (most recent call last): > File "sample_client.py", line 616, in > loop.run_until_complete(main_task) > ... > File "sample_client.py", line 319, in main > await actions[args.action](args, socket) > ... > File "sample_client.py", line 558, in sync_job > await asyncio.sleep(args.delay) > KeyboardInterrupt > > I recognize that the event loop mechanics are probably useful in some > circumstances, but in my experience working with asyncio for the last year, > I've found the stack traces are generally too noisy. For a while, I was > pasting stack traces into a text editor and cleaning them up by hand, but a > few months ago I wrote a custom excepthook that generates stack traces > similar to the one above. > > def async_excepthook(type_, exc, tb): > cause_exc = None > cause_str = None > > if exc.__cause__ is not None: > cause_exc = exc.__cause__ > cause_str = 'The above exception was the direct cause ' \ > 'of the following exception:' > elif exc.__context__ is not None and not exc.__suppress_context__: > cause_exc = exc.__context__ > cause_str = 'During handling of the above exception, ' \ > 'another exception occurred:' > > if cause_exc: > async_excepthook(type(cause_exc), cause_exc, > cause_exc.__traceback__) > > if cause_str: > print('\n{}\n'.format(cause_str)) > > print('Async Traceback (most recent call last):') > for frame in traceback.extract_tb(tb): > head, tail = os.path.split(frame.filename) > if (head.endswith('asyncio') or tail == 'traceback.py') and \ > frame.name.startswith('_'): > print(' ...') > continue > print(' File "{}", line {}, in {}' > .format(frame.filename, frame.lineno, frame.name)) > print(' {}'.format(frame.line)) > print('{}: {}'.format(type_.__name__, exc)) > > The meat of it is towards the bottom, "if > head.endswith('asyncio')..."There are a lot of debatable details and this > implementation is pretty hacky and clumsy, but I have found it valuable in > my own usage, and I haven't yet missed the omitted stack frames. > > I'm posting here to get constructive criticism on the concept and would > also like to hear if Curio or Trio have done anything like this. (Based on > a quick skim of the docs, I don't think they are.) I might publish a PyPI > package if anybody is interested. > > Cheers, > Mark > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ > -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From mehaase at gmail.com Tue Nov 14 10:27:30 2017 From: mehaase at gmail.com (Mark E. Haase) Date: Tue, 14 Nov 2017 10:27:30 -0500 Subject: [Async-sig] Simplifying stack traces for tasks? In-Reply-To: References: Message-ID: I'm not asking to change Python's default behavior. I'm asking if anybody else likes this idea, has ideas to make it better, and would use it if I published some form of it on PyPI. On Tue, Nov 14, 2017 at 10:08 AM, Andrew Svetlov wrote: > AFAIK Python never hides stdlib codelines in tracebacks. > Why we should start to do it in asyncio? > > On Tue, Nov 14, 2017 at 4:54 PM Mark E. Haase wrote: > >> If an exception is thrown while the `asyncio` event loop is running, the >> stack trace is pretty complicated. Here's an example: >> >> Traceback (most recent call last): >> File "sample_client.py", line 616, in >> loop.run_until_complete(main_task) >> File "/usr/lib/python3.6/asyncio/base_events.py", line 437, in >> run_until_complete >> self.run_forever() >> File "/usr/lib/python3.6/asyncio/base_events.py", line 405, in >> run_forever >> self._run_once() >> File "/usr/lib/python3.6/asyncio/base_events.py", line 1382, in >> _run_once >> handle._run() >> File "/usr/lib/python3.6/asyncio/events.py", line 126, in _run >> self._callback(*self._args) >> File "/usr/lib/python3.6/asyncio/tasks.py", line 315, in _wakeup >> self._step() >> File "/usr/lib/python3.6/asyncio/tasks.py", line 239, in _step >> result = coro.send(None) >> File "sample_client.py", line 319, in main >> await actions[args.action](args, socket) >> File "/usr/lib/python3.6/asyncio/coroutines.py", line 105, in >> __next__ >> return self.gen.send(None) >> File "sample_client.py", line 558, in sync_job >> await asyncio.sleep(args.delay) >> File "/usr/lib/python3.6/asyncio/coroutines.py", line 105, in >> __next__ >> return self.gen.send(None) >> File "/usr/lib/python3.6/asyncio/tasks.py", line 522, in sleep >> future, result) >> File "/usr/lib/python3.6/asyncio/base_events.py", line 525, in >> call_later >> timer = self.call_at(self.time() + delay, callback, *args) >> File "/usr/lib/python3.6/asyncio/base_events.py", line 541, in >> call_at >> timer = events.TimerHandle(when, callback, args, self) >> File "/usr/lib/python3.6/asyncio/events.py", line 148, in __init__ >> super().__init__(callback, args, loop) >> File "/usr/lib/python3.6/asyncio/events.py", line 92, in __init__ >> self._source_traceback = traceback.extract_stack(sys._ >> getframe(1)) >> File "/usr/lib/python3.6/traceback.py", line 207, in extract_stack >> stack = StackSummary.extract(walk_stack(f), limit=limit) >> File "/usr/lib/python3.6/traceback.py", line 352, in extract >> filename, lineno, name, lookup_line=False, locals=f_locals)) >> KeyboardInterrupt >> >> Most of the stack frames are inside asyncio (_run, _wakeup, _step) and >> reveal the mechanics of the event loop. The stack trace would be a lot >> easier to read (and more similar to a stack trace of an equivalent >> synchronous program) if it looked like this: >> >> Traceback (most recent call last): >> File "sample_client.py", line 616, in >> loop.run_until_complete(main_task) >> ... >> File "sample_client.py", line 319, in main >> await actions[args.action](args, socket) >> ... >> File "sample_client.py", line 558, in sync_job >> await asyncio.sleep(args.delay) >> KeyboardInterrupt >> >> I recognize that the event loop mechanics are probably useful in some >> circumstances, but in my experience working with asyncio for the last year, >> I've found the stack traces are generally too noisy. For a while, I was >> pasting stack traces into a text editor and cleaning them up by hand, but a >> few months ago I wrote a custom excepthook that generates stack traces >> similar to the one above. >> >> def async_excepthook(type_, exc, tb): >> cause_exc = None >> cause_str = None >> >> if exc.__cause__ is not None: >> cause_exc = exc.__cause__ >> cause_str = 'The above exception was the direct cause ' \ >> 'of the following exception:' >> elif exc.__context__ is not None and not exc.__suppress_context__: >> cause_exc = exc.__context__ >> cause_str = 'During handling of the above exception, ' \ >> 'another exception occurred:' >> >> if cause_exc: >> async_excepthook(type(cause_exc), cause_exc, >> cause_exc.__traceback__) >> >> if cause_str: >> print('\n{}\n'.format(cause_str)) >> >> print('Async Traceback (most recent call last):') >> for frame in traceback.extract_tb(tb): >> head, tail = os.path.split(frame.filename) >> if (head.endswith('asyncio') or tail == 'traceback.py') and \ >> frame.name.startswith('_'): >> print(' ...') >> continue >> print(' File "{}", line {}, in {}' >> .format(frame.filename, frame.lineno, frame.name)) >> print(' {}'.format(frame.line)) >> print('{}: {}'.format(type_.__name__, exc)) >> >> The meat of it is towards the bottom, "if head.endswith('asyncio')..."There >> are a lot of debatable details and this implementation is pretty hacky and >> clumsy, but I have found it valuable in my own usage, and I haven't yet >> missed the omitted stack frames. >> >> I'm posting here to get constructive criticism on the concept and would >> also like to hear if Curio or Trio have done anything like this. (Based on >> a quick skim of the docs, I don't think they are.) I might publish a PyPI >> package if anybody is interested. >> >> Cheers, >> Mark >> _______________________________________________ >> Async-sig mailing list >> Async-sig at python.org >> https://mail.python.org/mailman/listinfo/async-sig >> Code of Conduct: https://www.python.org/psf/codeofconduct/ >> > -- > Thanks, > Andrew Svetlov > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Tue Nov 14 11:40:02 2017 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Tue, 14 Nov 2017 16:40:02 +0000 Subject: [Async-sig] Simplifying stack traces for tasks? In-Reply-To: References: Message-ID: Got you. PyPI library makes sense. On Tue, Nov 14, 2017 at 5:27 PM Mark E. Haase wrote: > I'm not asking to change Python's default behavior. I'm asking if anybody > else likes this idea, has ideas to make it better, and would use it if I > published some form of it on PyPI. > > On Tue, Nov 14, 2017 at 10:08 AM, Andrew Svetlov > wrote: > >> AFAIK Python never hides stdlib codelines in tracebacks. >> Why we should start to do it in asyncio? >> >> On Tue, Nov 14, 2017 at 4:54 PM Mark E. Haase wrote: >> >>> If an exception is thrown while the `asyncio` event loop is running, the >>> stack trace is pretty complicated. Here's an example: >>> >>> Traceback (most recent call last): >>> File "sample_client.py", line 616, in >>> loop.run_until_complete(main_task) >>> File "/usr/lib/python3.6/asyncio/base_events.py", line 437, in >>> run_until_complete >>> self.run_forever() >>> File "/usr/lib/python3.6/asyncio/base_events.py", line 405, in >>> run_forever >>> self._run_once() >>> File "/usr/lib/python3.6/asyncio/base_events.py", line 1382, in >>> _run_once >>> handle._run() >>> File "/usr/lib/python3.6/asyncio/events.py", line 126, in _run >>> self._callback(*self._args) >>> File "/usr/lib/python3.6/asyncio/tasks.py", line 315, in _wakeup >>> self._step() >>> File "/usr/lib/python3.6/asyncio/tasks.py", line 239, in _step >>> result = coro.send(None) >>> File "sample_client.py", line 319, in main >>> await actions[args.action](args, socket) >>> File "/usr/lib/python3.6/asyncio/coroutines.py", line 105, in >>> __next__ >>> return self.gen.send(None) >>> File "sample_client.py", line 558, in sync_job >>> await asyncio.sleep(args.delay) >>> File "/usr/lib/python3.6/asyncio/coroutines.py", line 105, in >>> __next__ >>> return self.gen.send(None) >>> File "/usr/lib/python3.6/asyncio/tasks.py", line 522, in sleep >>> future, result) >>> File "/usr/lib/python3.6/asyncio/base_events.py", line 525, in >>> call_later >>> timer = self.call_at(self.time() + delay, callback, *args) >>> File "/usr/lib/python3.6/asyncio/base_events.py", line 541, in >>> call_at >>> timer = events.TimerHandle(when, callback, args, self) >>> File "/usr/lib/python3.6/asyncio/events.py", line 148, in __init__ >>> super().__init__(callback, args, loop) >>> File "/usr/lib/python3.6/asyncio/events.py", line 92, in __init__ >>> self._source_traceback = >>> traceback.extract_stack(sys._getframe(1)) >>> File "/usr/lib/python3.6/traceback.py", line 207, in extract_stack >>> stack = StackSummary.extract(walk_stack(f), limit=limit) >>> File "/usr/lib/python3.6/traceback.py", line 352, in extract >>> filename, lineno, name, lookup_line=False, locals=f_locals)) >>> KeyboardInterrupt >>> >>> Most of the stack frames are inside asyncio (_run, _wakeup, _step) and >>> reveal the mechanics of the event loop. The stack trace would be a lot >>> easier to read (and more similar to a stack trace of an equivalent >>> synchronous program) if it looked like this: >>> >>> Traceback (most recent call last): >>> File "sample_client.py", line 616, in >>> loop.run_until_complete(main_task) >>> ... >>> File "sample_client.py", line 319, in main >>> await actions[args.action](args, socket) >>> ... >>> File "sample_client.py", line 558, in sync_job >>> await asyncio.sleep(args.delay) >>> KeyboardInterrupt >>> >>> I recognize that the event loop mechanics are probably useful in some >>> circumstances, but in my experience working with asyncio for the last year, >>> I've found the stack traces are generally too noisy. For a while, I was >>> pasting stack traces into a text editor and cleaning them up by hand, but a >>> few months ago I wrote a custom excepthook that generates stack traces >>> similar to the one above. >>> >>> def async_excepthook(type_, exc, tb): >>> cause_exc = None >>> cause_str = None >>> >>> if exc.__cause__ is not None: >>> cause_exc = exc.__cause__ >>> cause_str = 'The above exception was the direct cause ' \ >>> 'of the following exception:' >>> elif exc.__context__ is not None and not >>> exc.__suppress_context__: >>> cause_exc = exc.__context__ >>> cause_str = 'During handling of the above exception, ' \ >>> 'another exception occurred:' >>> >>> if cause_exc: >>> async_excepthook(type(cause_exc), cause_exc, >>> cause_exc.__traceback__) >>> >>> if cause_str: >>> print('\n{}\n'.format(cause_str)) >>> >>> print('Async Traceback (most recent call last):') >>> for frame in traceback.extract_tb(tb): >>> head, tail = os.path.split(frame.filename) >>> if (head.endswith('asyncio') or tail == 'traceback.py') and \ >>> frame.name.startswith('_'): >>> print(' ...') >>> continue >>> print(' File "{}", line {}, in {}' >>> .format(frame.filename, frame.lineno, frame.name)) >>> print(' {}'.format(frame.line)) >>> print('{}: {}'.format(type_.__name__, exc)) >>> >>> The meat of it is towards the bottom, "if >>> head.endswith('asyncio')..."There are a lot of debatable details and this >>> implementation is pretty hacky and clumsy, but I have found it valuable in >>> my own usage, and I haven't yet missed the omitted stack frames. >>> >>> I'm posting here to get constructive criticism on the concept and would >>> also like to hear if Curio or Trio have done anything like this. (Based on >>> a quick skim of the docs, I don't think they are.) I might publish a PyPI >>> package if anybody is interested. >>> >>> Cheers, >>> Mark >>> _______________________________________________ >>> Async-sig mailing list >>> Async-sig at python.org >>> https://mail.python.org/mailman/listinfo/async-sig >>> Code of Conduct: https://www.python.org/psf/codeofconduct/ >>> >> -- >> Thanks, >> Andrew Svetlov >> > > -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rogerpate at gmail.com Tue Nov 14 17:00:57 2017 From: rogerpate at gmail.com (Roger Pate) Date: Tue, 14 Nov 2017 17:00:57 -0500 Subject: [Async-sig] Simplifying stack traces for tasks? In-Reply-To: References: Message-ID: On Tue, Nov 14, 2017 at 9:54 AM, Mark E. Haase wrote: ... > print('Async Traceback (most recent call last):') > for frame in traceback.extract_tb(tb): > head, tail = os.path.split(frame.filename) > if (head.endswith('asyncio') or tail == 'traceback.py') and \ > frame.name.startswith('_'): ... > The meat of it is towards the bottom, "if head.endswith('asyncio')..."There > are a lot of debatable details and this implementation is pretty hacky and > clumsy, but I have found it valuable in my own usage, and I haven't yet > missed the omitted stack frames. It would be better to determine if the qualified module name is "traceback" or starts with "asyncio." (or topmost package is "asyncio", etc.) rather than allow false positives for random_package.asyncio.module._any_function or random_package.traceback._any_function. I don't see an easy way to get the module object at this point in your hook; however: import asyncio, traceback excluded = [ asyncio.__file__[-len(os.path.basename(asyncio.__file__)):], # preserve existing separator to attempt being platform-agnostic traceback.__file__, ] ... for frame in traceback.extract_tb(tb): if not any(frame.filename.startswith(x) for x in excluded) and frame.name.startswith("_"): Functions starting with "_" in any module can now be excluded from tracebacks. I'm not fond of this implementation, but I hope it serves to illustrate. From njs at pobox.com Tue Nov 14 19:10:34 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 14 Nov 2017 16:10:34 -0800 Subject: [Async-sig] Simplifying stack traces for tasks? In-Reply-To: References: Message-ID: On Tue, Nov 14, 2017 at 6:54 AM, Mark E. Haase wrote: > If an exception is thrown while the `asyncio` event loop is running, the > stack trace is pretty complicated. Here's an example: > [...] > > I'm posting here to get constructive criticism on the concept and would also > like to hear if Curio or Trio have done anything like this. (Based on a > quick skim of the docs, I don't think they are.) I might publish a PyPI > package if anybody is interested. Trio does jump through hoops to try to make its tracebacks more readable, but so far this is mostly aimed at cases where there are multiple exceptions happening concurrently, and there is still an unpleasant amount of internal gibberish that can overwhelm the parts the user actually cares about. Doing something about this is on my todo list [1], and it shouldn't be *too* hard given that Trio already has to take over sys.excepthook, but I'm not 100% sure how to best track which frames are pure bookkeeping noise and which are interesting -- it's a bit more subtle than just "in the trio namespace or not". (I'd also like to annotate the traceback with more information, e.g. showing where an exception jumped between tasks.) Messing with tracebacks like this does feel a bit weird, but async libraries are in a weird position. The interpreter hides traceback lines all the time, because every time you go into C then the traceback disappears; Python's tracebacks are specifically designed to show user code, not interpreter internals. Async libraries are doing the same work the interpreter does, like implementing exception propagation machinery, but they're doing it in Python code, so it messes up the normal heuristic that C = low-level interpreter machinery, Python = user code. -n [1] https://github.com/python-trio/trio/issues/56 -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Tue Nov 14 19:15:45 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 14 Nov 2017 16:15:45 -0800 Subject: [Async-sig] Simplifying stack traces for tasks? In-Reply-To: References: Message-ID: On Tue, Nov 14, 2017 at 2:00 PM, Roger Pate wrote: > On Tue, Nov 14, 2017 at 9:54 AM, Mark E. Haase wrote: > ... >> print('Async Traceback (most recent call last):') >> for frame in traceback.extract_tb(tb): >> head, tail = os.path.split(frame.filename) >> if (head.endswith('asyncio') or tail == 'traceback.py') and \ >> frame.name.startswith('_'): > ... >> The meat of it is towards the bottom, "if head.endswith('asyncio')..."There >> are a lot of debatable details and this implementation is pretty hacky and >> clumsy, but I have found it valuable in my own usage, and I haven't yet >> missed the omitted stack frames. > > It would be better to determine if the qualified module name is > "traceback" or starts with "asyncio." (or topmost package is > "asyncio", etc.) rather than allow false positives for > random_package.asyncio.module._any_function or > random_package.traceback._any_function. I don't see an easy way to > get the module object at this point in your hook; however: You can't get the module from the cooked data that extract_tb returns, but it's there in the tb object itself. This walks the traceback and prints each frame's module: current = tb while current is not None: print("Next module", current.tb_frame.f_globals.get("__name__")) current = current.tb_next -n -- Nathaniel J. Smith -- https://vorpus.org