How pickle helps in reading huge files?

[Irmen de Jong]

On 16-10-2013 23:04, Peter Cacioppi wrote:
> On Tuesday, October 15, 2013 11:55:26 PM UTC-7, Harsh Jha wrote:
>> I've a huge csv file and I want to read stuff from it again and again. Is it useful
>> to pickle it and keep and then unpickle it whenever I need to use that data? Is it
>> faster that accessing that file simply by opening it again and again? Please
>> explain, why?
>> 
>> 
>> 
>> Thank you.
> 
> Surprising no-one else mentioned a fairly typical pattern for this sort of situation
> - the compromise between "read from disk" and "read from memory" is "implement a
> cache".

...or: use memory mapped I/O. Just let the OS deal with the 'caching' of memory pages.

Irmen