From dimaqq at gmail.com  Sat Jun  9 04:00:23 2018
From: dimaqq at gmail.com (Dima Tisnek)
Date: Sat, 9 Jun 2018 16:00:23 +0800
Subject: [Async-sig] async file IO library
Message-ID: <CAGGBzXJp8FD4jX_9s0i-L7FFU3Pa2wmtHRqsbfS=55M+O8=_hg@mail.gmail.com>

Hi group,

It seems that Python docs don't make a recommendation about which library
to use for asynchronous access to files when using asyncio.

Is there a canonical recommendation?
Is it a good idea?

There's at least one 3rd party library popular enough to deserve own
stackoverflow tag, however AFAIU it uses thread pool, while most OS'es
provide some or other API, making native implementation possible, at least
in theory.

What are your thoughts?

Thanks,
Dima
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20180609/d7d73951/attachment.html>

From njs at pobox.com  Sat Jun  9 07:24:01 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 9 Jun 2018 04:24:01 -0700
Subject: [Async-sig] async file IO library
In-Reply-To: <CAGGBzXJp8FD4jX_9s0i-L7FFU3Pa2wmtHRqsbfS=55M+O8=_hg@mail.gmail.com>
References: <CAGGBzXJp8FD4jX_9s0i-L7FFU3Pa2wmtHRqsbfS=55M+O8=_hg@mail.gmail.com>
Message-ID: <CAPJVwBm+C8yHaYEmq7uhi1DcgGDffKNNHpej2fzgZL3VT2RcGw@mail.gmail.com>

Using a thread pool is the right answer. There's some discussion here:
https://github.com/python-trio/trio/issues/20

One thing Python could do to make this more effective would be to add
a "buffer-only" mode for operations on file objects:
https://bugs.python.org/issue32561

-n

On Sat, Jun 9, 2018 at 1:00 AM, Dima Tisnek <dimaqq at gmail.com> wrote:
> Hi group,
>
> It seems that Python docs don't make a recommendation about which library to
> use for asynchronous access to files when using asyncio.
>
> Is there a canonical recommendation?
> Is it a good idea?
>
> There's at least one 3rd party library popular enough to deserve own
> stackoverflow tag, however AFAIU it uses thread pool, while most OS'es
> provide some or other API, making native implementation possible, at least
> in theory.
>
> What are your thoughts?
>
> Thanks,
> Dima
>
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>



-- 
Nathaniel J. Smith -- https://vorpus.org

From gmludo at gmail.com  Sun Jun 17 11:29:05 2018
From: gmludo at gmail.com (Ludovic Gasc)
Date: Sun, 17 Jun 2018 17:29:05 +0200
Subject: [Async-sig] Tips and tricks to track memory leaks in realtime
 inside an AsyncIO daemon ?
Message-ID: <CAON-fpFUu96tDwv1ymb1=PoqPdxw5uzyNNWUYqLRqYG7qXVZfQ@mail.gmail.com>

Hi,

We have now an AsyncIO daemon with memory leaks.

To track this, I'm thinking to use this:
https://pythonhosted.org/Pympler/muppy.html
and objgraph.

But, because it's a live daemon and not a script, instead to implement an
HTTP endpoint to launch the memory snapshot locally, I'm thinking to use
epdb:
https://github.com/sassoftware/epdb
To have directly a Python console to explore interactively inside the
process to have a more flexible way to debug.

I did a quick'n'dirty lab', it seems to work more or less to run epdb
inside a thread executor, but before to continue on this idea, I'm
interested in how you track your memory leaks inside your AsyncIO daemons ?

Thanks for your feedbacks.
--
Ludovic Gasc (GMLudo)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20180617/100895fb/attachment.html>

From alex.gronholm at nextday.fi  Mon Jun 18 03:52:50 2018
From: alex.gronholm at nextday.fi (alex.gronholm at nextday.fi)
Date: Mon, 18 Jun 2018 10:52:50 +0300
Subject: [Async-sig] Tips and tricks to track memory leaks in realtime
 inside an AsyncIO daemon ?
In-Reply-To: <CAON-fpFUu96tDwv1ymb1=PoqPdxw5uzyNNWUYqLRqYG7qXVZfQ@mail.gmail.com>
References: <CAON-fpFUu96tDwv1ymb1=PoqPdxw5uzyNNWUYqLRqYG7qXVZfQ@mail.gmail.com>
Message-ID: <68c0f9e2e517fb2af7a3b057235548fc422e7da7.camel@nextday.fi>

I've done something similar but using aioconsole instead of epdb.
su, 2018-06-17 kello 17:29 +0200, Ludovic Gasc kirjoitti:
> Hi,
> 
> We have now an AsyncIO daemon with memory leaks.
> 
> To track this, I'm thinking to use this:
> https://pythonhosted.org/Pympler/muppy.html
> and objgraph.
> 
> But, because it's a live daemon and not a script, instead to
> implement an HTTP endpoint to launch the memory snapshot locally, I'm
> thinking to use epdb:
> https://github.com/sassoftware/epdb
> To have directly a Python console to explore interactively inside the
> process to have a more flexible way to debug.
> 
> I did a quick'n'dirty lab', it seems to work more or less to run epdb
> inside a thread executor, but before to continue on this idea, I'm
> interested in how you track your memory leaks inside your AsyncIO
> daemons ?
> 
> Thanks for your feedbacks.
> --
> Ludovic Gasc (GMLudo)
> 
> _______________________________________________Async-sig mailing
> listAsync-sig at python.orghttps://mail.python.org/mailman/listinfo/asyn
> c-sigCode of Conduct: https://www.python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20180618/5aabfb0d/attachment.html>

From gmludo at gmail.com  Mon Jun 18 17:33:29 2018
From: gmludo at gmail.com (Ludovic Gasc)
Date: Mon, 18 Jun 2018 23:33:29 +0200
Subject: [Async-sig] Tips and tricks to track memory leaks in realtime
 inside an AsyncIO daemon ?
In-Reply-To: <68c0f9e2e517fb2af7a3b057235548fc422e7da7.camel@nextday.fi>
References: <CAON-fpFUu96tDwv1ymb1=PoqPdxw5uzyNNWUYqLRqYG7qXVZfQ@mail.gmail.com>
 <68c0f9e2e517fb2af7a3b057235548fc422e7da7.camel@nextday.fi>
Message-ID: <CAON-fpFPWXrKopZd0xnniTt+0VHLPDC=WZffhua54-ii8yquVg@mail.gmail.com>

Hi Alex,

Thanks for the tip, I will dig in it.

Have a nice week-end.
--
Ludovic Gasc (GMLudo)


Le lun. 18 juin 2018 ? 09:52, <alex.gronholm at nextday.fi> a ?crit :

> I've done something similar but using aioconsole instead of epdb.
>
> su, 2018-06-17 kello 17:29 +0200, Ludovic Gasc kirjoitti:
>
> Hi,
>
> We have now an AsyncIO daemon with memory leaks.
>
> To track this, I'm thinking to use this:
> https://pythonhosted.org/Pympler/muppy.html
> and objgraph.
>
> But, because it's a live daemon and not a script, instead to implement an
> HTTP endpoint to launch the memory snapshot locally, I'm thinking to use
> epdb:
> https://github.com/sassoftware/epdb
> To have directly a Python console to explore interactively inside the
> process to have a more flexible way to debug.
>
> I did a quick'n'dirty lab', it seems to work more or less to run epdb
> inside a thread executor, but before to continue on this idea, I'm
> interested in how you track your memory leaks inside your AsyncIO daemons ?
>
> Thanks for your feedbacks.
> --
> Ludovic Gasc (GMLudo)
>
> _______________________________________________
>
> Async-sig mailing list
>
> Async-sig at python.org
>
> https://mail.python.org/mailman/listinfo/async-sig
>
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20180618/167df4b0/attachment.html>