How pickle helps in reading huge files?

[Peter Cacioppi]

On Tuesday, October 15, 2013 11:55:26 PM UTC-7, Harsh Jha wrote:
> I've a huge csv file and I want to read stuff from it again and again. Is it useful to pickle it and keep and then unpickle it whenever I need to use that data? Is it faster that accessing that file simply by opening it again and again? Please explain, why?
> 
> 
> 
> Thank you.

Surprising no-one else mentioned a fairly typical pattern for this sort of situation - the compromise between "read from disk" and "read from memory" is "implement a cache".

I've had lots of good experiences hand rolling simple caches, especially if there is an application specific access pattern.

Python has nice implementations of things like tuple and dictionary which make caching fairly easy compared to other languages.