[SciPy-user] read/write compressed files

[David Warde-Farley]

On 20-Jun-07, at 4:18 PM, Dominik Szczerba wrote:

> I got it (partially) working, but am not sure about optimality. In
> particular, will fromstring copy memory into the array or  
> decompress in
> place? I think the former (how else would it know the size, and tell()
> will be slow), but please correct me if I am wrong.

I would almost certainly bet it would do a copy. Did you try using  
Anne's suggestion of scipy.read_array
with your 'fh' object?

Also, somebody correct me if I'm wrong, but I don't think modifying  
the 'shape' property directly is the
recommended way to do it, I think you should be using ps.resize().

David


> import gzip
> fh = gzip.GzipFile("test.dat.gz", 'rb');
> #ps = zeros(256*256) - will it help?
> ps = fromstring(fh.read(), 'd')
> ps.shape = (256,256)
> fh.close()
> fp = open('test.dat', 'wb')
> io.numpyio.fwrite(fp, ps.size, ps)
> fp.close()
>
> - Dominik
>
> Dominik Szczerba wrote:
>> That works very well for ascii files, but I failed to figure out  
>> about
>> binary data...
>> Thanks for any hints,
>> - Dominik
>>
>> Anne Archibald wrote:
>>> On 20/06/07, Dominik Szczerba <domi at vision.ee.ethz.ch> wrote:
>>>> Yes, I know it, but it does not return a scipy array, does it?
>>>> Can I achieve it without copying memory? (I have huge arrays to  
>>>> process)
>>> If the bz2 module will provide a file-like object, scipy.read_array
>>> can read from that.
>>>
>>> Anne
>>> _______________________________________________
>>> SciPy-user mailing list
>>> SciPy-user at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/scipy-user
>>
>
> -- 
> Dominik Szczerba, Ph.D.
> Computer Vision Lab CH-8092 Zurich
> http://www.vision.ee.ethz.ch/~domi
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user