best way to read a huge ascii file.

Marko Rauhamaa marko at pacujo.net
Fri Nov 25 10:29:44 EST 2016


Heli <hemla21 at gmail.com>:

> I have a huge ascii file(40G) and I have around 100M lines. I read this
> file using :
>
> [...]
>
> The problem is reading the file takes around 80 min while the
> interpolation only takes 15 mins.
>
> I was wondering if there is a more optimized way to read the file that
> would reduce the time to read the input file?

It looks pretty fast to me already; that's 20,000 lines per second, or
65 megabits per second.


Marko



More information about the Python-list mailing list