best way to read a huge ascii file.

[Chris Angelico]

On Thu, Dec 1, 2016 at 3:26 AM, BartC <bc at freeuk.com> wrote:
> On 30/11/2016 16:16, Heli wrote:
>>
>> Hi all,
>>
>>  Writing my ASCII file once to either of pickle or npy or hdf data types
>> and then working afterwards on the result binary file reduced the read time
>> from 80(min) to 2 seconds.
>
>
> 240,000% faster? Something doesn't sound quite right! How big is the file
> now? (The one that had been 40GB of text and contained 100M lines or
> datasets, although file caching is just about a possibility.)

Seems reasonable to me. Coming straight off the disk cache.

ChrisA