[Numpy-discussion] Efficient way to load a 1Gb file?

[Russell E. Owen]

In article 
<CA+RwOBWjyY_abjijnxEPkSeRaeom608uiMYwffGaG-6XDgSdPw at mail.gmail.com>,
 Torgil Svensson <torgil.svensson at gmail.com> wrote:

> Try the fromiter function, that will allow you to pass an iterator
> which can read the file line by line and not preload the whole file.
> 
> file_iterator = iter(open('filename.txt')
> line_parser = lambda x: map(float,x.split('\t'))
> a=np.fromiter(itertools.imap(line_parser,file_iterator),dtype=float)
> 
> You have also the option to iterate the file twice and pass the
> "count" argument.

Thanks. That sounds great!

-- RUssell