[Numpy-discussion] [Fwd: compression in storage of Numeric/numarray objects]
Joost van Evert
phjoost at gmail.com
Fri Sep 9 13:29:10 EDT 2005
On Fri, 2005-09-09 at 15:06 -0500, John Hunter wrote:
> >>>>> "Joost" == Joost van Evert <phjoost at gmail.com> writes:
>
> Joost> is it possible to use compression while storing
> Joost> numarray/Numeric objects?
>
>
> Sure
>
> In [35]: s = rand(10000)
>
> In [36]: file('uncompressed.dat', 'wb').write(s.tostring())
>
> In [37]: ls -l uncompressed.dat
> -rw-r--r-- 1 jdhunter jdhunter 80000 2005-09-09 15:04 uncompressed.dat
>
> In [38]: gzip.open('compressed.dat', 'wb').write(s.tostring())
>
> In [39]: ls -l compressed.dat
> -rw-r--r-- 1 jdhunter jdhunter 41393 2005-09-09 15:04 compressed.dat
>
Thanks, this helps me, but I think not enough, because the arrays I work
on are sometimes >1Gb(Correlation matrices). The tostring method would
explode the size, and result in a lot of swapping. Ideally the
compression also works with memmory mapped arrays.
Greets,
Joost
More information about the NumPy-Discussion
mailing list