best(fastest) way to send and get lists from files

[Yu-Xi Lim]

Abrahams, Max wrote:
> I've looked into pickle, dump, load, save, readlines(), etc.
> 
> Which is the best method? Fastest? My lists tend to be around a thousand to a million items.
> 
> Binary and text files are both okay, text would be preferred in general unless there's a significant speed boost from something binary.
> 
> thanks

1) Why don't you time them with the timeit module? 
http://docs.python.org/lib/module-timeit.html

Results will vary with the specific data you have, and your hardware 
speed, but if it's a lot of data, it's most likely going to be the 
latter that's the bottleneck. A compact binary format will help 
alleviate this.

If you're reading a lot of data into memory, you might have to deal with 
your OS swap/virtual memory.

2) "Best" depends on what your data is and what you're doing with it.

Are you reinventing a flat-file database? There are better solutions for 
databases.

If you're just reformatting data to pass to another program, say, for 
scientific computation, the portability may be more of an issue. Number 
crunching the resultant data may be even more time consuming such that 
the time spent writing/reading it becomes insignificant.