Writev

[Adam DePrince]

On Sun, 2004-12-19 at 23:43, Jp Calderone wrote:
> On Sun, 19 Dec 2004 23:12:27 -0500, Adam DePrince <adam at cognitcorp.com> wrote:
> > [snip]
> > 
> > Of course, to take advantage of this requires that writev be exposed.  I
> > have an implementation of writev.  This implementation is reasonably
> > smart, it "unrolls" only so as many iteration.next calls as necessary to
> > fill the pointer list for the next invocation of writev.   Of course, on
> > many systems (Linux, BSD) writev's limit is 1024 items, so to
> > accommodate users who don't want to peel off and hold in memory 1024
> > iteration.next()'s, you can set an optional parameter to a smaller
> > value. 
> > 
> > I'm not sure where to take this as a "next step."  It seems too small a
> > change for a PEP.  Any ideas?
> > 
> > You can download the patch from
> > http://deprince.net/software/writev/index.html
> > 
> 
>   writev() in the standard library would be nice.  I have an 
> implementation too ;)  I wonder how many other people do as 
> well.
> 
>   Regarding the implementation, just one part confuses me.  
> What's the idea behind this check at the beginning of the 
> iterator unrolling loop?
> 
> +    if ( !PyObject_CheckReadBuffer( buf ) )
> +      continue;
> 
>   Is it me, or is that silently ignoring bad input?
> 
>   Other miscellaneous feedback:
> 
>     The use of alloca() isn't common in CPython.  PyMem_Malloc 
> is probably what you want to be calling.  At the very least, 
> alloca()'s return value should be checked to be sure it is 
> non-NULL.  If you switch to PyMem_Malloc, you'll have to be 
> sure to free the memory, of course.
> 
>     The support of iterators is a cool idea, but I'm not sure
> it is actually useful.  Consider the case where not all bytes 
> are written.  How is an application supposed to handle writing 
> the rest of the bytes, if they have been consumed from an 
> iterator?  Hopefully I'm missing the obvious answer, because
> supporting arbitrary iterators would be cool, but it seems like
> only sequences can really be usefully supported.
> 

Fixed this.  The return value is a tuple of bytes written and a tuple of
stuff that didn't get shipped.  And yes, the first element is replaced
with a string that represents the fraction of that element that didn't
get sent.   Here is simple sample code that illustrates how this works.

data = [ .... lots of little strings in this list ... ]
data = iter( data )
import itertools
import writev

while 1:
  (count, leftovers ) = writev.writev( fd, data )
  if leftovers == None: break
  data = itertools.chain( iter( leftovers ), data )
  ... Of course, our loop would in a real example be controlled by a
select.  In this naive example we would want to sleep ... 

Adam DePrince