[Numpy-discussion] Re: memory-mapped Numeric arrays: arrayfrombuffer version 2

[Kragen Sitaker]

Paul Dubois writes:
> I have verified that this package seems to work on Windows. I says seems
> only because I didn't try enough to uncover anything subtle.

Thanks!  If you run maparray.py from the command-line, it runs a basic
regression test suite, which doesn't really try enough to uncover
anything subtle either.

> Unless or until we are convinced as a community that this is (a) the
> right way to do this and (b) that the package is portable, it would not
> be wise to put it in the main distribution. 

Well, I'm not the community, but I'll state my arguments.

On (a): I don't know about the right way to do it; as I'm sure is
obvious, I'm new to extending Numerical Python in C, but I doubt
there's a simpler way to do it ("it" being seeing the contents of
files as Numeric arrays), and I think it does work as well as it's
possible to get it to work without major hacking of Numeric (to
support read-only arrays) or the mmap module (to prevent closing an
open object, to allow read-only mmapping on Windows).  The other
things on the wishlist are basically features to add --- "start" and
"size" arguments to maparray(), a "create-a-file" argument to
maparray(), etc., and don't change the basic structure.

On (b): it depends on two-argument mmap.mmap(), open(),
<file>.fileno(), and os.fstat().  The major portability hurdle there
is probably mmap.mmap(), but that's OK.

> I would like to hear from the community about this so that I will know
> whether or not to add this package as a separate SourceForge 'package'
> within the Numerical Python area. Meantime I will add a link to the web
> page.

I would too.  There have been downloads from something like 50 IP
addresses, but I've only heard from three people.