[PYTHON MATRIX-SIG] Re: Saving HUGE arrays

[Guido van Rossum]

> One solution would be to add some nice stream protocol to Python.
> Files would be one kind of stream, memory blocks another one.
> Then file.write(memory_block) should do the following:
> 
> 1) The file object asks the memory block object for the next
>    chunk of data (however large), imposing a maximum (which
>    would actually be infinite in case of files).
> 
> 2) The memory block object provides a pointer and length indicator for
>    such a block and remembers where to continue, or an end-of-stream
>    indication when there is no more data.
> 
> 3) The file object writes the data to the disk file.
> 
> 4) Goto 1.
> 
> Such a protocol would transfer data in the largest blocks that both
> objects can handle.
>   
> > Next try (though this is getting complicated :-( ): how about a
> > "sequence-of-buffers" interface.  E.g.:
> 
> I am not too fond of solutions that just try to work around one
> special problem. But of course it's better than raising exceptions
> for arrays.

My proposal exactly matches your item (2), so we may be on to
something...

--Guido van Rossum (home page: http://www.python.org/~guido/)

=================
MATRIX-SIG  - SIG on Matrix Math for Python

send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================