[Numpy-discussion] RE: memory-mapped Numeric arrays: arrayfrombuffer version 2

[Paul F. Dubois]

I have verified that this package seems to work on Windows. I says seems
only because I didn't try enough to uncover anything subtle.

Unless or until we are convinced as a community that this is (a) the
right way to do this and (b) that the package is portable, it would not
be wise to put it in the main distribution. 

I would like to hear from the community about this so that I will know
whether or not to add this package as a separate SourceForge 'package'
within the Numerical Python area. Meantime I will add a link to the web
page.

-----Original Message-----
From: python-announce-list-admin at python.org
[mailto:python-announce-list-admin at python.org] On Behalf Of Kragen
Sitaker
Sent: Wednesday, January 23, 2002 9:40 PM
To: python-announce-list at python.org
Subject: memory-mapped Numeric arrays: arrayfrombuffer version 2


The 'arrayfrombuffer' package features support for Numerical Python
arrays whose contents are stored in buffer objects, including
memory-mapped files.  This has the following advantages:

- loading your array from a file is easy --- a module import and a
  single function call --- and doesn't use excessive amounts of
  memory.
- loading your array is quick; it doesn't need to be copied from one
  part of memory to another in order to be loaded.
- your array gets demand-loaded; parts you aren't using don't need to
  be in memory or in swap.
- under memory-pressure conditions, your array doesn't use up swap,
  and parts of it you haven't modified can be evicted from RAM without
  the need for a disk write
- your arrays can be bigger than your physical memory
- when you modify your array, only the parts you modify get written
  back out to disk

This is something that's been requested on the Numpy list a few times a
year since 1999.

arrayfrombuffer lives at http://pobox.com/~kragen/sw/arrayfrombuffer/
The current version is version 2; it is released under the X11 license
(the BSD license without the advertising clause).

<kragen at pobox.com>

<P><A
HREF="http://pobox.com/~kragen/sw/arrayfrombuffer/">arrayfrombuffer
2</A> - creates Numeric arrays from memory-mapped files.  (23-Jan-02)

-- 
http://mail.python.org/mailman/listinfo/python-announce-list