[Python-ideas] An alternate approach to async IO

[Trent Nelson]

On Tue, Nov 27, 2012 at 12:13:18PM -0800, Richard Oudkerk wrote:
> On 27/11/2012 7:59pm, Trent Nelson wrote:
> >> >But you have to allocate the buffer*before*  you initiate an overlapped
> >> >read.  And you may as well make that buffer a Python bytes object (which
> >> >can be shrunk if it is too large).  That leaves no "processing" that can
> >> >usefully be done by a C level thread pool.
> >      I'm a little confused by that last sentence.  The premise of my idea
> >      is being able to service AIO via simple GIL-independent threads that
> >      really just copy data from A to B.  The simple fact that they don't
> >      have to acquire the GIL each time the completion port has an event
> >      seems like a big win, no?
> >
> >      (So I'm not sure if you're saying that this wouldn't work because
> >      you may as well use Python bytes objects, and they can't be accessed
> >      willy-nilly from non-GIL threads... or if you're saying they can,
> >      but there's no benefit from a C-level thread copying data to/from
> >      buffers independent of the GIL.)
> >
> 
> I am saying that there is no copying necessary if you use a bytes object 
> as your buffer.  You can just use _PyBytes_Resize() afterwards to shrink 
> it if necessary.

    Got it.  So what about the "no processing that can be usefully done
    by a C level thread" bit?  I'm trying to discern whether or not you're
    highlighting a fundamental flaw in the theory/idea ;-)

    (That it's going to be more optimal to have background threads service
     IO without the need to acquire the GIL, basically.)

        Trent.