[SciPy-User] format for chunked file save and read ?

[josef.pktd at gmail.com]

On Wed, Sep 22, 2010 at 11:52 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Wed, Sep 22, 2010 at 10:41,  <josef.pktd at gmail.com> wrote:
>> On Wed, Sep 22, 2010 at 11:12 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>> On Wed, Sep 22, 2010 at 10:03,  <josef.pktd at gmail.com> wrote:
>>>> On Wed, Sep 22, 2010 at 10:36 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>>>> On Wed, Sep 22, 2010 at 09:18,  <josef.pktd at gmail.com> wrote:
>>>>>> What is the best file format for storing temporary data, for chunked
>>>>>> saving and loading, that only uses numpy and scipy?
>>>>>
>>>>> An mmapped .npy file would probably serve the purpose well enough.
>>>>
>>>> Thanks, I will try.
>>>>
>>>> The old stats.models had one example with memmapped temporary storage,
>>>> where I never managed to figure out how to find and close an open file
>>>> handle (on Windows) that was not cleaned up on delete or when the
>>>> instance went out of scope.
>>>
>>> Use numpy.lib.format.open_memmap(). When the mmapped array goes out of
>>> scope, the associated file object will also get finalized.
>>
>> I will have to experiment more systematically. Now I have several
>> memmapped files in my working directory that windows blocks as
>> accessed, and I'm not able to close them until I shut down the python
>> shell (which I don't want to do since I have other work in it.)
>
> Are you using IPython? IPython can hold onto references to many of the
> objects that you use.

I'm back to POI, plain old IDLE

Josef

>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>