[SciPy-dev] Difficulty creating an example for illustrating memmap.flush()

[David Goldsmith]

Hi, David.  Thanks for your (validating) reply. :-)  Well, there's something more going on than just the size of the mmap or the size of the change to the mmap: I experimented with adding a shape (10,) complex128 array to shape (10, 10), (10, 10, 10), (10, 10, 10, 10), (10, 10, 10, 10, 10) mmaps (and their corresponding "flat" mmaps, shape (100,), etc.), both just in the last dimension using [9, 9, 9, 9, :], e.g., and everywhere using [:,:,:,:,:] - which worked! - and the file on disc was updated every time without ever having to call flush, not once!  So, hopefully, someone who knows what's going on will chime in...

DG

--- On Sun, 8/2/09, David Gowers <00ai99 at gmail.com> wrote:

> From: David Gowers <00ai99 at gmail.com>
> Subject: Re: [SciPy-dev] Difficulty creating an example for illustrating memmap.flush()
> To: "SciPy Developers List" <scipy-dev at scipy.org>
> Date: Sunday, August 2, 2009, 4:56 PM
> 
> 
> On Mon, Aug 3, 2009 at 7:30 AM,
> David Goldsmith <d_l_goldsmith at yahoo.com>
> wrote:
> 
> 
> 
> Hi, folks.  So, I'm trying to devise an example to
> illustrate memmap.flush().  It didn't work the way I
> was intuitively expecting (i.e., it didn't appear to do
> anything at all) so I went looking for a
> "ready-made" example and found:
> 
> 
> 
> 
> http://www.slideshare.net/enthought/python-for-scientific-computing-webinar-may-22-2009
> 
> 
> 
> which has an example on Slide 25.  However, here's
> what I get when I try to duplicate the example:
> 
> 
> 
> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC
> v.1310 32 bit
> 
> win32
> 
> >>> import numpy as np
> 
> >>> np.version.version
> 
> '1.3.0rc2'
> 
> >>> q =
> np.memmap('new_file.dat',mode='w+',shape=(2,5))
> 
> >>> q
> 
> memmap([[0, 0, 0, 0, 0],
> 
>        [0, 0, 0, 0, 0]], dtype=uint8)
> 
> >>> # Print out underlying file contents
> 
> ... # Note: not using iPython, so have to use os.system
> 
> ...
> 
> >>> import os
> 
> >>> os.system('type new_file.dat')
> 
>           0
> 
> >>> # Note: already a little different than
> Webinar Expl.
> 
> ...
> 
> >>> # Next write ascii value for 'A' (65)
> into q
> 
> ...
> 
> >>> q[:] = ord('A')
> 
> >>> q
> 
> memmap([[65, 65, 65, 65, 65],
> 
>        [65, 65, 65, 65, 65]], dtype=uint8)
> 
> >>> # Do I need to call flush before file is
> written to?
> 
> ...
> 
> >>> os.system('type new_file.dat')
> 
> AAAAAAAAAA0
> 
> >>> # No! Does flushing change anything?
> 
> ...
> 
> >>> q.flush()
> 
> >>> os.system('type new_file.dat')
> 
> AAAAAAAAAA0
> 
> >>> # No! Is it because I printed q before
> checking it on disc?
> 
> ...
> 
> >>> # Start afresh, but don't print memmap
> before checking it
> 
> ...
> 
> >>> r =
> np.memmap('new_file2.dat',mode='w+',shape=(2,5))
> 
> >>> os.system('type new_file2.dat') #
> "reproducibility check"
> 
>           0
> 
> >>> r[:] = ord('A')
> 
> >>> os.system('type new_file2.dat') #
> Checking file on disc immediately
> 
> AAAAAAAAAA0
> 
> >>> # File is updated without calling flush,
> indeed without any
> 
> ... # intervening access to the memmap at all!
> 
> 
> 
> What gives?  Is this a bug?  If not, can someone please
> furnish me with an example that clearly an explicitly
> illustrates the function (and necessity) of memmap.flush()?
> Sorry, I can't. I can say that I
> encountered this when I first discovered np.memmap and began
> using it for a lot of things. At one point, I didn't
> seem to need to flush; In other cases I did.
> 
> 
> I wonder whether it has to do with the underlying i/o
> buffering (eg. if your array is smaller than the buffer, all
> changes are written immediately)
> 
> David
> 
> 
> -----Inline Attachment Follows-----
> 
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>