Creating Long Lists

[Ben Finney]

Kelson Zawack <zawackkfb at gis.a-star.edu.sg> writes:

> I have a large (10gb) data file for which I want to parse each line
> into an object and then append this object to a list for sorting and
> further processing.

What is the nature of the further processing?

Does that further processing access the items sequentially? If so, they
don't all need to be in memory at once, and you can produce them with a
generator <URL:http://docs.python.org/glossary.html#term-generator>.

Note that, if you just want lines of text from a file, the file object
itself is a generator for the lines of text within it.

If, on the other hand, you need arbitrary access all over that large
data set, you probably want a data type better suited. The standard
library has the ‘array’ module for this purpose; the third-party NumPy
library provides even more power.

-- 
 \       “Remember: every member of your ‘target audience’ also owns a |
  `\   broadcasting station. These ‘targets’ can shoot back.” —Michael |
_o__)               Rathbun to advertisers, news.admin.net-abuse.email |
Ben Finney