[Numpy-discussion] Reading a big netcdf file

[Gökhan Sever]

On Wed, Aug 3, 2011 at 3:15 PM, Christopher Barker <Chris.Barker at noaa.gov>wrote:

> On 8/3/11 1:57 PM, Gökhan Sever wrote:
> > This is what I get here:
> >
> > In [1]: a = np.zeros((21601, 10801), dtype=np.uint16)
> >
> > In [2]: a.tofile('temp.npa')
> >
> > In [3]: del a
> >
> > In [4]: timeit a = np.fromfile('temp.npa', dtype=np.uint16)
> > 1 loops, best of 3: 251 ms per loop
>
> so that's about 10 times faster than my machine. I didn't think disks
> had gotten much faster -- they are still generally 7200 rpm (or slower
> in laptops).
>
> So I've either got a really slow disk, or you have a really fast one (or
> both), or maybe you're getting cache effect, as you wrote the file just
> before reading it.
>
> repeating, doing just what you did:
>
> In [8]: timeit a = np.fromfile('temp.npa', dtype=np.uint16)
> 1 loops, best of 3: 2.53 s per loop
>
> then I wrote a bunch of others to disk, and tried again:
>
> In [17]: timeit a = np.fromfile('temp.npa', dtype=np.uint16)
> 1 loops, best of 3: 2.45 s per loop
>
> so ti seems I'm not seeing cache effects, but maybe you are.
>
> Anyway, we haven't heard from the OP -- I'm not sure what s/he thought
> was slow.
>
> -Chris



In [11]: a = np.zeros((21601, 10801), dtype=np.uint16)

In [12]: a.tofile('temp.npa')

In [13]: del a

Quitting here and restarting IPython. (this should cut the caching effect
isn't it?)

I[1]: timeit a = np.fromfile('temp.npa', dtype=np.uint16)
1 loops, best of 3: 263 ms per loop

#More information about my system:
hdparm -I /dev/sda | grep Rotation
Nominal Media Rotation Rate: 7200

uname -a  #64-bit Fedora 14
Linux ccn 2.6.35.13-92.fc14.x86_64 #1

Filesystem(s) ext4

-- 
Gökhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110803/ba3dad9c/attachment.html>