From dimaqq at gmail.com Thu Feb 2 10:36:08 2017 From: dimaqq at gmail.com (Dima Tisnek) Date: Thu, 2 Feb 2017 16:36:08 +0100 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world Message-ID: Hi list, sorry for being a bit late, but I only discovered that post recently... I've a couple concerns: 1. update now 3.6 is out, the post should be rewritten or new one made, mainly because this post if what inquisitive minds find... 2. asyncio for whom? I've presented asyncio tutorial of sourts at a local Python meetup. I've noticed that asyncio documentation is pretty hard to understand. There are many concepts and a ton of caveats. I now finally understand why, and I think it comes down to asyncio being 2 distinct things: a. asyncio is a standard library meant to be used as a framework by other, higher-level, 3rd party libraries that are still written by very smart developers, and b. asyncio is a way for average joe developers to write async code. I used to think I was in cat.a., but clearly I'm more of a cat.b. dev :( I really with there was decent advice how to use asyncio/aiohttp/pytest-asyncio/curio? etc. Something along the lines of: * if you want fire-and-forget (e.g. async logging), do xyz1 * if you want a worker task, do xyz2 * if you want a synchronisation point, do xyz3 * if you want ... I'd love these to be reflected in the API that Python advertises. For example, I'd rather think of a Future() as a thing that provides a result, meaning it's commonly shared by multiple tasks, meaning it would by default be shielded. Should a Future() die if it's garbage-collected, like a generator or should it conitnue? These are areas where Python should provide best practice. Task cancellation (just like thread cancellation) is by definition(?) arcane, and I'd argue average developer should be advised against... I for one, still can't make up my mind what's better -- to cancel bottom or top frame when task being cancelled awaits something that awaits something else, etc... Should provision to block cancellation even exist? What would be the best quorum for these discussions? Cheers, d. P.S. I'm trying to avoid "coroutine" nomenclature on purpose From mehaase at gmail.com Thu Feb 2 14:40:40 2017 From: mehaase at gmail.com (Mark E. Haase) Date: Thu, 2 Feb 2017 14:40:40 -0500 Subject: [Async-sig] Debugging destroyed, pending tasks Message-ID: I'm still climbing up the asyncio learning curve. I'm building an application on top of asyncio, and one of my goals is that when the application is interrupted (e.g. SIGINT), it should carefully clean up all running tasks before closing the event loop. Some tasks should be cancelled right away, but other tasks should be allowed to complete. As an example of the latter, I may have several file download tasks that I want to finish before closing the event loop. (If I get a 2nd SIGINT, then I close the loop immediately and exit.) I have a hard time debugging the "Task was destroyed..." errors. For example: Task was destroyed but it is pending! task: wait_for= cb=[_wait.._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]> I'm pretty sure that I understand why I get this error: this is a task (e.g. created with asyncio.ensure_future()) that was never awaited. To prevent this error, I should await the task (or call its result() method, right?). But how do I figure out where the task was created? I have PYTHONASYNCIODEBUG=1 in my environment, and I see the "defined at" text, which is helpful if the coroutine is in my own code, but for asyncio.sleep(), it's not very helpful. Is it possible to see where this task was created, e.g. the line containing asyncio.ensure_future(asyncio.sleep(...)) ? If not, is this a capability that could be added to debug mode? E.g. ensure_future would add its own stack trace to the task object so that it could be retrieved later when a pending task is destroyed. In my current scenario, I have few enough tasks that there's only one place I think this particular error could come from, but as my application grows, this will get harder and harder to debug. Thoughts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov at gmail.com Thu Feb 2 14:53:03 2017 From: yselivanov at gmail.com (Yury Selivanov) Date: Thu, 2 Feb 2017 14:53:03 -0500 Subject: [Async-sig] Debugging destroyed, pending tasks In-Reply-To: References: Message-ID: > On Feb 2, 2017, at 2:40 PM, Mark E. Haase wrote: > > Is it possible to see where this task was created, e.g. the line containing asyncio.ensure_future(asyncio.sleep(...)) ? If not, is this a capability that could be added to debug mode? E.g. ensure_future would add its own stack trace to the task object so that it could be retrieved later when a pending task is destroyed. > Yeah, this is doable. I suggest you to open an issue on bugs.python.org. Yury From mehaase at gmail.com Thu Feb 2 16:35:54 2017 From: mehaase at gmail.com (Mark E. Haase) Date: Thu, 2 Feb 2017 16:35:54 -0500 Subject: [Async-sig] Debugging destroyed, pending tasks In-Reply-To: References:

Message-ID: Thanks, Yury, for the quick reply, but I just realized my question was based on my own mistake. Asyncio already does exactly what I want, but I have a subprocess that wasn't inheriting PYTHONASYNCIODEBUG=1 from its parent. Once I've exported this environment variable to the subprocess, I am now seeing stack traces that show where pending tasks were created. Sorry everybody! Please ignore this thread... On Thu, Feb 2, 2017 at 2:53 PM, Yury Selivanov wrote: > > > On Feb 2, 2017, at 2:40 PM, Mark E. Haase wrote: > > > > Is it possible to see where this task was created, e.g. the line > containing asyncio.ensure_future(asyncio.sleep(...)) ? If not, is this a > capability that could be added to debug mode? E.g. ensure_future would add > its own stack trace to the task object so that it could be retrieved later > when a pending task is destroyed. > > > > Yeah, this is doable. I suggest you to open an issue on bugs.python.org. > > Yury -------------- next part -------------- An HTML attachment was scrubbed... URL: