processing a Very Large file

[Gregory Bond]

DJTB wrote:
> Hi,
> 
> I'm trying to manually parse a dataset stored in a file. The data should be
> converted into Python objects.

In addition to what the others have mentioned, this sort of problem is 
pretty easy to do with a C coded extension type, if you have (or can 
buy/borrow) any C skills. The result is waaaay more efficient in time 
and memory, particularly if you actually don't wind up looking at most 
elements.

But Robert's solution (use iterators/generators rather than store them 
all in a list) is probably best/quickest if that is possible for your app.